Jul 29 16:49:12.576843 ip-10-0-139-76 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 16:49:12.576857 ip-10-0-139-76 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 16:49:12.576868 ip-10-0-139-76 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 16:49:12.577193 ip-10-0-139-76 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 16:49:22.689830 ip-10-0-139-76 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 16:49:22.689848 ip-10-0-139-76 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5c1a78843eb64496aaac65f09001377f --
Jul 29 16:51:22.041336 ip-10-0-139-76 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 16:51:22.397368 ip-10-0-139-76 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 16:51:22.397368 ip-10-0-139-76 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 16:51:22.397368 ip-10-0-139-76 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 16:51:22.397368 ip-10-0-139-76 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 16:51:22.397368 ip-10-0-139-76 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 16:51:22.399295 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.399205    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 16:51:22.406959 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406936    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:22.406959 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406957    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:22.406959 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406963    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:22.406959 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406966    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406969    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406972    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406975    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406978    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406981    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406983    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406986    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406988    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406991    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406994    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406996    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.406999    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407002    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407005    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407007    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407010    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407012    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407015    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407017    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:22.407142 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407020    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407022    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407025    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407029    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407033    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407037    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407040    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407042    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407045    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407056    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407058    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407061    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407064    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407067    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407070    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407073    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407076    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407096    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407101    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407105    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:22.407624 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407109    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407111    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407114    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407117    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407119    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407122    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407124    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407127    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407130    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407133    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407136    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407138    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407141    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407143    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407146    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407149    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407152    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407154    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407157    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407160    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:22.408166 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407162    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407166    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407168    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407172    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407176    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407179    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407182    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407185    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407188    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407190    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407193    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407196    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407198    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407201    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407203    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407206    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407208    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407211    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407213    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407216    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:22.408648 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407219    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407222    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407224    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407226    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407229    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407231    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407234    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407236    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407239    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407241    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407244    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407246    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407249    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407251    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407254    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407256    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407940    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407946    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407950    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407953    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:22.409152 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407956    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407959    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407962    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407964    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407967    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407969    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407972    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407975    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407977    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407980    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407982    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407986    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407991    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407994    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407997    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.407999    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408002    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408005    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408007    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:22.409739 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408010    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408012    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408015    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408017    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408019    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408022    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408024    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408027    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408030    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408032    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408035    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408038    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408041    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408045    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408048    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408052    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408054    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408058    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408060    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408063    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:22.410231 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408066    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408068    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408071    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408073    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408076    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408093    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408096    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408099    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408102    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408105    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408107    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408110    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408112    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408116    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408118    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408121    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408123    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408126    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408128    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408131    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:22.410725 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408133    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408136    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408138    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408142    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408146    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408148    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408151    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408154    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408156    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408159    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408161    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408164    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408167    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408170    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408172    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408176    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408179    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408181    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408184    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408186    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:22.411249 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408189    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408192    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408195    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408197    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408200    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408203    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408205    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408208    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408210    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408213    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408216    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408218    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408221    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408223    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408227    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408231    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408606    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408612    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408615    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:22.411740 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408618    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408621    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408624    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408627    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408630    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408632    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408635    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408639    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408642    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408645    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408648    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408651    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408654    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408657    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408660    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408663    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408667    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408670    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408673    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:22.412210 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408676    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408678    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408681    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408683    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408686    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408688    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408691    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408693    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408695    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408699    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408701    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408704    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408706    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408709    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408712    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408714    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408717    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408723    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408726    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408728    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:22.412672 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408731    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408733    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408737    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408741    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408743    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408746    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408749    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408752    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408755    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408758    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408761    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408764    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408767    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408770    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408772    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408775    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408777    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408780    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408783    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408785    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:22.413170 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408790    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408793    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408796    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408798    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408801    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408803    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408806    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408808    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408811    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408813    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408816    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408818    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408821    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408823    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408826    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408829    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408832    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408835    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408837    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408840    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:22.413651 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408842    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408845    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408847    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408850    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408852    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408855    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408857    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408860    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408862    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408865    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408867    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408870    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408872    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408876    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408878    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408881    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.408883    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408951    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408958    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408963    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408967    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 16:51:22.414141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408971    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408974    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408978    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408982    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408985    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408988    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408991    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408995    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.408998    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409000    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409003    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409006    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409010    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409012    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409017    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409020    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409023    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409026    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409029    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409033    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409036    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409038    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409042    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409047    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 16:51:22.414642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409050    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409053    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409056    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409059    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409063    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409066    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409069    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409072    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409075    2559 flags.go:64] FLAG: --enable-server="true"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409091    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409096    2559 flags.go:64] FLAG: --event-burst="100"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409099    2559 flags.go:64] FLAG: --event-qps="50"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409102    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409104    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409107    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409111    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409114    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409117    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409120    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409123    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409126    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409129    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409132    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409135    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409138    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 16:51:22.415223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409141    2559 flags.go:64] FLAG: --feature-gates=""
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409144    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409147    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409150    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409153    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409157    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409161    2559 flags.go:64] FLAG: --help="false"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409165    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409168    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409171    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409174    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409178    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409181    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409184    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409187    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409190    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409193    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409196    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409199    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409202    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409205    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409207    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409210    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409213    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 16:51:22.415874 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409216    2559 flags.go:64] FLAG: --lock-file=""
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409218    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409221    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409224    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409229    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409233    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409236    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409239    2559 flags.go:64] FLAG: --logging-format="text"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409242    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409245    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409248    2559 flags.go:64] FLAG: --manifest-url=""
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409250    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409259    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409262    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409267    2559 flags.go:64] FLAG: --max-pods="110"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409270    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409273    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409276    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409279    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409282    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409286    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409288    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409295    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409298    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409301    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 16:51:22.416458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409304    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409307    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409312    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409315    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409318    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409321    2559 flags.go:64] FLAG: --port="10250"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409324    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409327    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d9ab45db27b26b8"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409330    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409333    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409335    2559 flags.go:64] FLAG: --register-node="true"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409338    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409342    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409345    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409348    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409351    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409354    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409357    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409360    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409365    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409368    2559 flags.go:64] FLAG: --runonce="false"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409373    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409376    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409379    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409382    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 16:51:22.417060 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409385    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409388    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409391    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409395    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409398    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409401    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409403    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409406    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409410    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409413    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409416    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409421    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409424    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409427    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409431    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409434    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409437    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409439    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409442    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409445    2559 flags.go:64] FLAG: --v="2"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409449    2559 flags.go:64] FLAG: --version="false"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409453    2559 flags.go:64] FLAG: --vmodule=""
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409457    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.409460    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409575    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:22.417685 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409579    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409585    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409588    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409592    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409596    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409598    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409601    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409604    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409607    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409610    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409613    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409615    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409618    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409621    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409623    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409626    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409628    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409631    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409633    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409636    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:22.418325 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409639    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409643    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409646    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409649    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409652    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409655    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409658    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409661    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409663    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409666    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409669    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409671    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409674    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409678    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409680    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409686    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409688    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409691    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409693    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409696    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:22.418799 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409699    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409703    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409706    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409709    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409712    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409715    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409718    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409720    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409723    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409726    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409729    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409734    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409736    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409739    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409742    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409745    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409748    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409751    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409754    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:22.419340 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409756    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409759    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409762    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409765    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409767    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409770    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409774    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409776    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409780    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409783    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409785    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409788    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409790    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409793    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409796    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409798    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409800    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409803    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409806    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409808    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:22.419808 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409811    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409813    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409816    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409818    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409821    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409824    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409826    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409828    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409831    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409834    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409836    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409839    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409841    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409844    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409847    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409849    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409852    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409855    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:22.420339 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.409859    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.410510    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.417304    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.417323    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.417893    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.417900    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417960    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417964    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417968    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417971    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417974    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417977    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417980    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417984    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417988    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:22.420789 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417992    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417995    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.417998    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418001    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418004    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418006    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418009    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418011    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418014    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418016    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418019    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418021    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418024    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418027    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418029    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418032    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418034    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418037    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418039    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418042    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:22.421175 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418051    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418054    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418057    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418060    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418062    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418065    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418067    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418070    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418072    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418075    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418089    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418092    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418095    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418098    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418101    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418103    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418106    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418109    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418111    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418114    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:22.421686 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418117    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418119    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418122    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418125    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418128    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418131    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418133    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418136    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418139    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418141    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418144    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418147    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418149    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418158    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418161    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418164    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418168    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418171    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418173    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:22.422190 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418176    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418179    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418182    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418185    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418187    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418190    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418193    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418196    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418198    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418201    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418204    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418207    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418210    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418212    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418215    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418217    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418220    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418223    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418226    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:22.422637 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418229    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418233    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418236    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418239    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418241    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418244    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418246    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418249    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418259    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418262    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418264    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418267    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.418272    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418410    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418414    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 16:51:22.423207 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418417    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418420    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418423    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418425    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418428    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418430    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418433    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418436    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418439    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418441    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418444    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418446    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418449    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418453    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418457    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418460    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418463    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418466    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418469    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 16:51:22.423622 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418471    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418474    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418476    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418479    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418482    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418484    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418486    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418495    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418497    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418500    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418502    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418505    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418507    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418510    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418512    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418515    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418517    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418520    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418522    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418525    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 16:51:22.424064 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418527    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418530    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418532    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418535    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418537    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418540    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418543    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418545    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418548    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418551    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418553    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418556    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418560    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418562    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418565    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418568    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418572    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418575    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 16:51:22.424567 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418578    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418581    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418589    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418592    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418595    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418597    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418600    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418603    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418605    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418608    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418610    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418613    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418615    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418618    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418620    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418623    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418625    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418628    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418630    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418633    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 16:51:22.425011 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418636    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418638    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418640    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418643    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418646    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418649    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418651    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418653    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418656    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418658    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418661    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418663    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418666    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418668    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418670    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418678    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418681    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418684    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418686    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 16:51:22.425492 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:22.418688    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 16:51:22.425969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.418694    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 16:51:22.425969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.418827    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 16:51:22.425969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.423358    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 16:51:22.425969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.424898    2559 server.go:1019] "Starting client certificate rotation"
Jul 29 16:51:22.425969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.425006    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 16:51:22.425969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.425033    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 16:51:22.428836 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.428822    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 16:51:22.435213 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.435194    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 16:51:22.446774 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.446753    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 29 16:51:22.449221 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.449204    2559 log.go:25] "Validated CRI v1 image API"
Jul 29 16:51:22.450122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.450096    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 16:51:22.454900 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.454845    2559 fs.go:135] Filesystem UUIDs: map[65720ed5-b368-41d2-8ed6-66a86e2567df:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ec439cea-11cd-4204-9e1d-8e93c12e614d:/dev/nvme0n1p3]
Jul 29 16:51:22.454900 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.454867    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 16:51:22.459538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.459425    2559 manager.go:216] Machine: {Timestamp:2026-07-29 16:51:22.458938175 +0000 UTC m=+0.308971864 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101828 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ba5407a8319c89293e0ac71d495bb SystemUUID:ec2ba540-7a83-19c8-9293-e0ac71d495bb BootID:5c1a7884-3eb6-4496-aaac-65f09001377f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:94:39:e9:ee:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:94:39:e9:ee:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:ee:63:63:e1:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 16:51:22.459538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.459529    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 16:51:22.459704 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.459602    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 16:51:22.462794 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.462766    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 16:51:22.462958 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.462797    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-76.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 16:51:22.463587 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.463575    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 16:51:22.463642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.463593    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 16:51:22.463642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.463603    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 16:51:22.464295 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.464283    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 16:51:22.465052 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.465036    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 16:51:22.466054 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.466041    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 16:51:22.470588 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470574    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 16:51:22.470722 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470710    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 16:51:22.470777 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470750    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 16:51:22.470777 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470764    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 16:51:22.470871 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470789    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 16:51:22.470871 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470803    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 29 16:51:22.470871 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.470815    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 16:51:22.471218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.471205    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 16:51:22.471293 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.471228    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 16:51:22.473327 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.473306    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 16:51:22.476569 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.476549    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 16:51:22.476645 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.476588    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 16:51:22.478500 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478487    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478504    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478510    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478516    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478525    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478533    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478543    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 16:51:22.478550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478549    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 16:51:22.478730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478556    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 16:51:22.478730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478562    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 16:51:22.478730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478570    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 16:51:22.478730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478575    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 16:51:22.478730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478599    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 16:51:22.478730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.478605    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 16:51:22.481774 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.481760    2559 server.go:1268] "Started kubelet"
Jul 29 16:51:22.481878 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.481850    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 16:51:22.482000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.481949    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 16:51:22.482065 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.482021    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 16:51:22.482670 ip-10-0-139-76 systemd[1]: Started Kubernetes Kubelet.
Jul 29 16:51:22.483097 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.483068    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 16:51:22.483476 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.483444    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 16:51:22.485112 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.485093    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-76.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 16:51:22.487576 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.487547    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 16:51:22.487576 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.487557    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 16:51:22.488591 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.488575    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 16:51:22.492176 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.492157    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 16:51:22.492176 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.492168    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 16:51:22.492316 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.492244    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:22.492925 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.492757    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 16:51:22.492925 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.492897    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 16:51:22.492925 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.492910    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 16:51:22.493116 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493045    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 16:51:22.493116 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493057    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 16:51:22.493342 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493325    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 16:51:22.493342 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493343    2559 factory.go:55] Registering systemd factory
Jul 29 16:51:22.493454 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493352    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 29 16:51:22.493649 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493636    2559 factory.go:153] Registering CRI-O factory
Jul 29 16:51:22.493716 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493651    2559 factory.go:223] Registration of the crio container factory successfully
Jul 29 16:51:22.493716 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493670    2559 factory.go:103] Registering Raw factory
Jul 29 16:51:22.493716 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.493683    2559 manager.go:1196] Started watching for new ooms in manager
Jul 29 16:51:22.494504 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.494422    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8ftsx"
Jul 29 16:51:22.494817 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.494779    2559 manager.go:349] Starting recovery of all containers
Jul 29 16:51:22.494882 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.493880    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-76.ec2.internal.18c6d0cd4d3d7fbb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-76.ec2.internal,UID:ip-10-0-139-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-76.ec2.internal,},FirstTimestamp:2026-07-29 16:51:22.481729467 +0000 UTC m=+0.331763157,LastTimestamp:2026-07-29 16:51:22.481729467 +0000 UTC m=+0.331763157,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-76.ec2.internal,}"
Jul 29 16:51:22.506122 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.506070    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 16:51:22.506398 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.506370    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8ftsx"
Jul 29 16:51:22.506868 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.506842    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 16:51:22.507943 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.507924    2559 manager.go:354] Recovery completed
Jul 29 16:51:22.512272 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.512260    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.513960 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.513944    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.514042 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.513977    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.514042 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.513993    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.514543 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.514530    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 16:51:22.514543 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.514541    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 16:51:22.514635 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.514556    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 16:51:22.514635 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.514593    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.515356 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.515341    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.515456 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.515367    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.515456 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.515377    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.517119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.517107    2559 policy_none.go:49] "None policy: Start"
Jul 29 16:51:22.517166 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.517121    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 16:51:22.517166 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.517131    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 16:51:22.517351 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.517280    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-76.ec2.internal.18c6d0cd4f294e0c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-76.ec2.internal,UID:ip-10-0-139-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-76.ec2.internal,},FirstTimestamp:2026-07-29 16:51:22.51396046 +0000 UTC m=+0.363994156,LastTimestamp:2026-07-29 16:51:22.51396046 +0000 UTC m=+0.363994156,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-76.ec2.internal,}"
Jul 29 16:51:22.519176 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.519165    2559 policy_none.go:47] "Start"
Jul 29 16:51:22.552617 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.552604    2559 manager.go:337] "Starting Device Plugin manager"
Jul 29 16:51:22.552678 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.552630    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 16:51:22.552678 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.552640    2559 server.go:87] "Starting device plugin registration server"
Jul 29 16:51:22.552865 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.552853    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 16:51:22.552896 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.552870    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 16:51:22.553003 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.552985    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 16:51:22.553104 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.553075    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 16:51:22.553104 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.553104    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 16:51:22.553449 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.553431    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 16:51:22.553531 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.553473    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:22.654498 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.653350    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.654498 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.654114    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.654498 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.654162    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.654498 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.654180    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.654498 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.654207    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.663515 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.663500    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.663581 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.663522    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-76.ec2.internal\": node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:22.697087 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.697064    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 16:51:22.697186 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.697137    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:22.698284 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.698270    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 16:51:22.698347 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.698303    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 16:51:22.698347 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.698322    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 16:51:22.698415 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.698353    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 16:51:22.707549 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.707529    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:22.797937 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.797778    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:22.798956 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.798939    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"]
Jul 29 16:51:22.799005 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.798997    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.800467 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.800454    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.800555 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.800479    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.800555 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.800489    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.802816 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.802802    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.802946 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.802930    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.803003 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.802967    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.803496 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.803478    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.803589 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.803506    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.803589 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.803517    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.803589 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.803485    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.803589 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.803586    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.803736 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.803599    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.805780 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.805765    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.805834 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.805794    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 16:51:22.806428 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.806413    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 16:51:22.806490 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.806439    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 16:51:22.806490 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.806448    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeHasSufficientPID"
Jul 29 16:51:22.835891 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.835871    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-76.ec2.internal\" not found" node="ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.840443 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.840427    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-76.ec2.internal\" not found" node="ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.895112 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.895079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6556c24d7e1dda4410307c41d995cf82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal\" (UID: \"6556c24d7e1dda4410307c41d995cf82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.895169 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.895126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6556c24d7e1dda4410307c41d995cf82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal\" (UID: \"6556c24d7e1dda4410307c41d995cf82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.895169 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.895155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfca4edb4041480c1eb9593a15543d37-config\") pod \"kube-apiserver-proxy-ip-10-0-139-76.ec2.internal\" (UID: \"bfca4edb4041480c1eb9593a15543d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.898792 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.898767    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:22.996150 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.996056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6556c24d7e1dda4410307c41d995cf82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal\" (UID: \"6556c24d7e1dda4410307c41d995cf82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.996150 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.996109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6556c24d7e1dda4410307c41d995cf82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal\" (UID: \"6556c24d7e1dda4410307c41d995cf82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.996308 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.996158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6556c24d7e1dda4410307c41d995cf82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal\" (UID: \"6556c24d7e1dda4410307c41d995cf82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.996308 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.996206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfca4edb4041480c1eb9593a15543d37-config\") pod \"kube-apiserver-proxy-ip-10-0-139-76.ec2.internal\" (UID: \"bfca4edb4041480c1eb9593a15543d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.996308 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.996233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6556c24d7e1dda4410307c41d995cf82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal\" (UID: \"6556c24d7e1dda4410307c41d995cf82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.996308 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:22.996235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfca4edb4041480c1eb9593a15543d37-config\") pod \"kube-apiserver-proxy-ip-10-0-139-76.ec2.internal\" (UID: \"bfca4edb4041480c1eb9593a15543d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:22.999427 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:22.999408    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.100489 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:23.100449    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.140260 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.140225    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:23.146145 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.145953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:23.201374 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:23.201342    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.302277 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:23.302204    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.403292 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:23.403262    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.425091 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.425044    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 16:51:23.425243 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.425224    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 16:51:23.503615 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:23.503584    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.513299 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.513267    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 16:46:22 +0000 UTC" deadline="2028-04-01 22:35:57.177483761 +0000 UTC"
Jul 29 16:51:23.513299 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.513296    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14693h44m33.664191692s"
Jul 29 16:51:23.604657 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:23.604564    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-76.ec2.internal\" not found"
Jul 29 16:51:23.644628 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.644602    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:23.660958 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.660934    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:23.693083 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.693053    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:23.708005 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.707988    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 16:51:23.709870 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.709849    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal"
Jul 29 16:51:23.725373 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.725351    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 16:51:23.749792 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.749768    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:23.803711 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:23.803684    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6556c24d7e1dda4410307c41d995cf82.slice/crio-fb7b321ca544069791c4cece1f9e891a8a330cf2b154ae26ff736d82ab5576dd WatchSource:0}: Error finding container fb7b321ca544069791c4cece1f9e891a8a330cf2b154ae26ff736d82ab5576dd: Status 404 returned error can't find the container with id fb7b321ca544069791c4cece1f9e891a8a330cf2b154ae26ff736d82ab5576dd
Jul 29 16:51:23.807277 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:23.807262    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 16:51:23.891254 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:23.891228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfca4edb4041480c1eb9593a15543d37.slice/crio-7246812edcc1a4928fd93a7b5c46ead494ce5c76b26345a2f6c58b928219e601 WatchSource:0}: Error finding container 7246812edcc1a4928fd93a7b5c46ead494ce5c76b26345a2f6c58b928219e601: Status 404 returned error can't find the container with id 7246812edcc1a4928fd93a7b5c46ead494ce5c76b26345a2f6c58b928219e601
Jul 29 16:51:24.474542 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.474517    2559 apiserver.go:52] "Watching apiserver"
Jul 29 16:51:24.484304 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.484275    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 16:51:24.485642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.485608    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal","openshift-cluster-node-tuning-operator/tuned-s2gzd","openshift-multus/multus-additional-cni-plugins-4w4tf","openshift-multus/multus-vc2dr","openshift-multus/network-metrics-daemon-stljd","openshift-network-operator/iptables-alerter-fk5zw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2","openshift-dns/node-resolver-jn7cv","openshift-image-registry/node-ca-lvfl4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal","openshift-network-diagnostics/network-check-target-ntfhp","openshift-ovn-kubernetes/ovnkube-node-scjn5","kube-system/global-pull-secret-syncer-gkqgp","kube-system/konnectivity-agent-dmbwc"]
Jul 29 16:51:24.488430 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.488408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:24.488522 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.488484    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:24.490794 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.490773    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.493212 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.493189    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 16:51:24.493212 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.493218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.494009 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.493990    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.494145 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.494053    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.494145 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.494054    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4f6z8\""
Jul 29 16:51:24.495780 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.495627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.496149 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.496128    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.497682 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.497661    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.498077 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.498045    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.498228 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.498135    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 16:51:24.498228 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.498156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 16:51:24.498916 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.498730    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lzmrr\""
Jul 29 16:51:24.499245 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.499103    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.499381 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.499364    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 16:51:24.499649 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.499625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d8vjg\""
Jul 29 16:51:24.499737 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.499718    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.501005 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.500541    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 16:51:24.501005 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.500571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.501411 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.501391    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4d56\""
Jul 29 16:51:24.503179 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.503112    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.503606 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.503588    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.503851 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.503831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.504053 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.504034    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 16:51:24.504210 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.504191    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tjzmz\""
Jul 29 16:51:24.505606 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.505585    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.506062 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08906a19-3edf-49f2-8a85-6fbf739bbb2e-tmp-dir\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.506166 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnbf\" (UniqueName: \"kubernetes.io/projected/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-kube-api-access-pfnbf\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.506166 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-hostroot\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506166 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-multus-certs\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506333 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:24.506333 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ps7\" (UniqueName: \"kubernetes.io/projected/08906a19-3edf-49f2-8a85-6fbf739bbb2e-kube-api-access-68ps7\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.506333 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.506333 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.506333 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-system-cni-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-cni-bin\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506420    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08906a19-3edf-49f2-8a85-6fbf739bbb2e-hosts-file\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-cni-multus\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-daemon-config\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506520    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-socket-dir-parent\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-kubelet\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.506607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-cni-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-cnibin\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-k8s-cni-cncf-io\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-conf-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblnh\" (UniqueName: \"kubernetes.io/projected/5912cfcc-c880-49bd-90f2-86d4034e5d4d-kube-api-access-tblnh\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5912cfcc-c880-49bd-90f2-86d4034e5d4d-cni-binary-copy\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-etc-kubernetes\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6jb\" (UniqueName: \"kubernetes.io/projected/f1c52131-cd16-415a-a195-0d47aca1a9c4-kube-api-access-xl6jb\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.506962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-os-release\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.507020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-netns\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.507274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.507145    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.507852 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.507731    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgfjp\""
Jul 29 16:51:24.510806 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.509746    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.510806 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.510042    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 16:51:24.510806 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.510088    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.510806 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.510401    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6nhkp\""
Jul 29 16:51:24.510806 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.510660    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 16:51:24.511129 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.510781    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:24.511129 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.511002    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:24.513142 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.513121    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.515583 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.515556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 16:51:24.515945 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.515923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.516057 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.516004    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:24.516917 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.516888    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 16:51:24.517055 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.516980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 16:51:24.518313 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.518292    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlwpm\""
Jul 29 16:51:24.518410 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.518327    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 16:51:24.518410 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.518333    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 16:51:24.518521 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.518433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.518724 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.518705    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 16:51:24.521333 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.521307    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 16:51:24.521440 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.521381    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 16:51:24.521699 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.521683    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mq7gs\""
Jul 29 16:51:24.538997 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.538974    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-stxmv"
Jul 29 16:51:24.549190 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.549166    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-stxmv"
Jul 29 16:51:24.594207 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.594179    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 16:51:24.607601 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-run-netns\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.607772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.607772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb01582e-77d0-49e4-9241-c3ab3a17cb59-tmp\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.607772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.607772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-kubelet\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.607772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-cni-netd\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.607772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-cni-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-cnibin\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-conf-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblnh\" (UniqueName: \"kubernetes.io/projected/5912cfcc-c880-49bd-90f2-86d4034e5d4d-kube-api-access-tblnh\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysctl-d\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-node-log\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5912cfcc-c880-49bd-90f2-86d4034e5d4d-cni-binary-copy\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-cnibin\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.607981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-sys\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-tuned\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.608102 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-cni-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-log-socket\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-os-release\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-netns\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-conf-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-os-release\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-var-lib-kubelet\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-netns\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9dbe303-c874-4a64-9bb0-b3da59e4ce8f-konnectivity-ca\") pod \"konnectivity-agent-dmbwc\" (UID: \"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f\") " pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81cd7021-f3ac-471d-a272-c8ff4492fef5-serviceca\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08906a19-3edf-49f2-8a85-6fbf739bbb2e-tmp-dir\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-hostroot\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.608662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbm8\" (UniqueName: \"kubernetes.io/projected/81cd7021-f3ac-471d-a272-c8ff4492fef5-kube-api-access-mwbm8\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5912cfcc-c880-49bd-90f2-86d4034e5d4d-cni-binary-copy\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-hostroot\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f51ad380-8ca6-47ae-be18-66d23531f26f-dbus\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ps7\" (UniqueName: \"kubernetes.io/projected/08906a19-3edf-49f2-8a85-6fbf739bbb2e-kube-api-access-68ps7\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08906a19-3edf-49f2-8a85-6fbf739bbb2e-tmp-dir\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608840    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-system-cni-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-cni-bin\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-os-release\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63c66261-984e-43ea-a1be-f63406b79d11-iptables-alerter-script\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.608967    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.608969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-host\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.609045    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:25.109022585 +0000 UTC m=+2.959056276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:24.609404 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9dbe303-c874-4a64-9bb0-b3da59e4ce8f-agent-certs\") pod \"konnectivity-agent-dmbwc\" (UID: \"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f\") " pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-system-cni-dir\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-var-lib-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-run-ovn-kubernetes\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-cni-bin\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609263    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-system-cni-dir\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609353    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-cni-bin\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77r4\" (UniqueName: \"kubernetes.io/projected/b4437f1a-045a-455c-9be1-92a486a48ab6-kube-api-access-f77r4\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08906a19-3edf-49f2-8a85-6fbf739bbb2e-hosts-file\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08906a19-3edf-49f2-8a85-6fbf739bbb2e-hosts-file\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-daemon-config\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-ovnkube-config\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-socket-dir-parent\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-kubelet\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysconfig\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45f8\" (UniqueName: \"kubernetes.io/projected/eb01582e-77d0-49e4-9241-c3ab3a17cb59-kube-api-access-l45f8\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-cnibin\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4rv\" (UniqueName: \"kubernetes.io/projected/ad7706df-e587-4d12-b151-37f895e4eeab-kube-api-access-bd4rv\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-systemd\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-k8s-cni-cncf-io\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-etc-kubernetes\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-kubernetes\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.609965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysctl-conf\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c66261-984e-43ea-a1be-f63406b79d11-host-slash\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-daemon-config\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-etc-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4437f1a-045a-455c-9be1-92a486a48ab6-ovn-node-metrics-cert\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-k8s-cni-cncf-io\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.610885 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6jb\" (UniqueName: \"kubernetes.io/projected/f1c52131-cd16-415a-a195-0d47aca1a9c4-kube-api-access-xl6jb\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-systemd\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-lib-modules\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-etc-kubernetes\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-systemd-units\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-multus-socket-dir-parent\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f51ad380-8ca6-47ae-be18-66d23531f26f-kubelet-config\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnbf\" (UniqueName: \"kubernetes.io/projected/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-kube-api-access-pfnbf\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-kubelet\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-multus-certs\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-ovnkube-script-lib\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-run-multus-certs\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.611760 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-modprobe-d\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxspj\" (UniqueName: \"kubernetes.io/projected/63c66261-984e-43ea-a1be-f63406b79d11-kube-api-access-fxspj\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81cd7021-f3ac-471d-a272-c8ff4492fef5-host\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-slash\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.610992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-ovn\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.611034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.611115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-env-overrides\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.611146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-cni-multus\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.611171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-run\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.612629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.611201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5912cfcc-c880-49bd-90f2-86d4034e5d4d-host-var-lib-cni-multus\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.618852 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.618803    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 16:51:24.624263 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.624212    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblnh\" (UniqueName: \"kubernetes.io/projected/5912cfcc-c880-49bd-90f2-86d4034e5d4d-kube-api-access-tblnh\") pod \"multus-vc2dr\" (UID: \"5912cfcc-c880-49bd-90f2-86d4034e5d4d\") " pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.624263 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.624221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ps7\" (UniqueName: \"kubernetes.io/projected/08906a19-3edf-49f2-8a85-6fbf739bbb2e-kube-api-access-68ps7\") pod \"node-resolver-jn7cv\" (UID: \"08906a19-3edf-49f2-8a85-6fbf739bbb2e\") " pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.624263 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.624235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6jb\" (UniqueName: \"kubernetes.io/projected/f1c52131-cd16-415a-a195-0d47aca1a9c4-kube-api-access-xl6jb\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:24.624470 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.624450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnbf\" (UniqueName: \"kubernetes.io/projected/5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d-kube-api-access-pfnbf\") pod \"aws-ebs-csi-driver-node-hdss2\" (UID: \"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.703607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.703544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal" event={"ID":"6556c24d7e1dda4410307c41d995cf82","Type":"ContainerStarted","Data":"fb7b321ca544069791c4cece1f9e891a8a330cf2b154ae26ff736d82ab5576dd"}
Jul 29 16:51:24.705005 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.704967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal" event={"ID":"bfca4edb4041480c1eb9593a15543d37","Type":"ContainerStarted","Data":"7246812edcc1a4928fd93a7b5c46ead494ce5c76b26345a2f6c58b928219e601"}
Jul 29 16:51:24.712414 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-cni-netd\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712534 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysctl-d\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712534 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-node-log\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712534 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-node-log\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712534 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-cni-netd\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712534 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-sys\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-tuned\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-log-socket\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-var-lib-kubelet\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-sys\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9dbe303-c874-4a64-9bb0-b3da59e4ce8f-konnectivity-ca\") pod \"konnectivity-agent-dmbwc\" (UID: \"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f\") " pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysctl-d\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-var-lib-kubelet\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-log-socket\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81cd7021-f3ac-471d-a272-c8ff4492fef5-serviceca\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbm8\" (UniqueName: \"kubernetes.io/projected/81cd7021-f3ac-471d-a272-c8ff4492fef5-kube-api-access-mwbm8\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f51ad380-8ca6-47ae-be18-66d23531f26f-dbus\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-os-release\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.712880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63c66261-984e-43ea-a1be-f63406b79d11-iptables-alerter-script\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-host\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9dbe303-c874-4a64-9bb0-b3da59e4ce8f-agent-certs\") pod \"konnectivity-agent-dmbwc\" (UID: \"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f\") " pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-system-cni-dir\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-var-lib-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.712991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-run-ovn-kubernetes\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-cni-bin\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713069    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-host\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713293    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81cd7021-f3ac-471d-a272-c8ff4492fef5-serviceca\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9dbe303-c874-4a64-9bb0-b3da59e4ce8f-konnectivity-ca\") pod \"konnectivity-agent-dmbwc\" (UID: \"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f\") " pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-var-lib-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-run-ovn-kubernetes\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-system-cni-dir\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77r4\" (UniqueName: \"kubernetes.io/projected/b4437f1a-045a-455c-9be1-92a486a48ab6-kube-api-access-f77r4\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-cni-bin\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.713650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-os-release\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-ovnkube-config\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysconfig\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f51ad380-8ca6-47ae-be18-66d23531f26f-dbus\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63c66261-984e-43ea-a1be-f63406b79d11-iptables-alerter-script\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45f8\" (UniqueName: \"kubernetes.io/projected/eb01582e-77d0-49e4-9241-c3ab3a17cb59-kube-api-access-l45f8\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-cnibin\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4rv\" (UniqueName: \"kubernetes.io/projected/ad7706df-e587-4d12-b151-37f895e4eeab-kube-api-access-bd4rv\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysconfig\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-systemd\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-cnibin\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-kubernetes\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysctl-conf\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-systemd\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-kubernetes\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c66261-984e-43ea-a1be-f63406b79d11-host-slash\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.714530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-etc-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4437f1a-045a-455c-9be1-92a486a48ab6-ovn-node-metrics-cert\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-sysctl-conf\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-systemd\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c66261-984e-43ea-a1be-f63406b79d11-host-slash\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-lib-modules\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad7706df-e587-4d12-b151-37f895e4eeab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-systemd\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-systemd-units\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-systemd-units\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.713988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f51ad380-8ca6-47ae-be18-66d23531f26f-kubelet-config\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-ovnkube-script-lib\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-etc-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f51ad380-8ca6-47ae-be18-66d23531f26f-kubelet-config\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-lib-modules\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-ovnkube-config\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.715402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-modprobe-d\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxspj\" (UniqueName: \"kubernetes.io/projected/63c66261-984e-43ea-a1be-f63406b79d11-kube-api-access-fxspj\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.714220    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81cd7021-f3ac-471d-a272-c8ff4492fef5-host\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-slash\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-ovn\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-env-overrides\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-modprobe-d\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-run\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-run-netns\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb01582e-77d0-49e4-9241-c3ab3a17cb59-tmp\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-kubelet\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81cd7021-f3ac-471d-a272-c8ff4492fef5-host\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-kubelet\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.716218 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714658    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-slash\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714677    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-ovnkube-script-lib\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.714700    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:51:25.214676205 +0000 UTC m=+3.064709900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb01582e-77d0-49e4-9241-c3ab3a17cb59-run\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714746    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-openvswitch\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-host-run-netns\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.714943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b4437f1a-045a-455c-9be1-92a486a48ab6-run-ovn\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.715064    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b4437f1a-045a-455c-9be1-92a486a48ab6-env-overrides\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.715253    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad7706df-e587-4d12-b151-37f895e4eeab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.717648 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.717600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4437f1a-045a-455c-9be1-92a486a48ab6-ovn-node-metrics-cert\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.717648 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.717626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb01582e-77d0-49e4-9241-c3ab3a17cb59-etc-tuned\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.719473 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.719439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb01582e-77d0-49e4-9241-c3ab3a17cb59-tmp\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.720606 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.720551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9dbe303-c874-4a64-9bb0-b3da59e4ce8f-agent-certs\") pod \"konnectivity-agent-dmbwc\" (UID: \"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f\") " pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.722141 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.722123    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:24.722242 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.722146    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:24.722242 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.722160    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:24.722351 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:24.722249    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:25.222230142 +0000 UTC m=+3.072263835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:24.724238 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.724194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbm8\" (UniqueName: \"kubernetes.io/projected/81cd7021-f3ac-471d-a272-c8ff4492fef5-kube-api-access-mwbm8\") pod \"node-ca-lvfl4\" (UID: \"81cd7021-f3ac-471d-a272-c8ff4492fef5\") " pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.725198 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.725126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77r4\" (UniqueName: \"kubernetes.io/projected/b4437f1a-045a-455c-9be1-92a486a48ab6-kube-api-access-f77r4\") pod \"ovnkube-node-scjn5\" (UID: \"b4437f1a-045a-455c-9be1-92a486a48ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.726123 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.726101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4rv\" (UniqueName: \"kubernetes.io/projected/ad7706df-e587-4d12-b151-37f895e4eeab-kube-api-access-bd4rv\") pod \"multus-additional-cni-plugins-4w4tf\" (UID: \"ad7706df-e587-4d12-b151-37f895e4eeab\") " pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.726987 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.726953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45f8\" (UniqueName: \"kubernetes.io/projected/eb01582e-77d0-49e4-9241-c3ab3a17cb59-kube-api-access-l45f8\") pod \"tuned-s2gzd\" (UID: \"eb01582e-77d0-49e4-9241-c3ab3a17cb59\") " pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.727692 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.727103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxspj\" (UniqueName: \"kubernetes.io/projected/63c66261-984e-43ea-a1be-f63406b79d11-kube-api-access-fxspj\") pod \"iptables-alerter-fk5zw\" (UID: \"63c66261-984e-43ea-a1be-f63406b79d11\") " pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.803025 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.802979    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jn7cv"
Jul 29 16:51:24.812211 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.812185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2"
Jul 29 16:51:24.813555 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.813523    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08906a19_3edf_49f2_8a85_6fbf739bbb2e.slice/crio-fb1ccbf26d90c8bec7dfc72ed03c6be950cd474073be352fb404deff2a6b0183 WatchSource:0}: Error finding container fb1ccbf26d90c8bec7dfc72ed03c6be950cd474073be352fb404deff2a6b0183: Status 404 returned error can't find the container with id fb1ccbf26d90c8bec7dfc72ed03c6be950cd474073be352fb404deff2a6b0183
Jul 29 16:51:24.823943 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.823909    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be0b3c8_8dd8_4ef3_b4d1_b25fd3647e5d.slice/crio-1025884b4a86b13965fc2a51addcec51f50c993d822b257e1aca710a7777b8cc WatchSource:0}: Error finding container 1025884b4a86b13965fc2a51addcec51f50c993d822b257e1aca710a7777b8cc: Status 404 returned error can't find the container with id 1025884b4a86b13965fc2a51addcec51f50c993d822b257e1aca710a7777b8cc
Jul 29 16:51:24.824534 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.824459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vc2dr"
Jul 29 16:51:24.828549 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.828525    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4w4tf"
Jul 29 16:51:24.836825 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.836801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fk5zw"
Jul 29 16:51:24.840592 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.840551    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7706df_e587_4d12_b151_37f895e4eeab.slice/crio-9febbed8a89d5286462a62c5ab89b1ff45985e847d698d0c93da28f4a764208c WatchSource:0}: Error finding container 9febbed8a89d5286462a62c5ab89b1ff45985e847d698d0c93da28f4a764208c: Status 404 returned error can't find the container with id 9febbed8a89d5286462a62c5ab89b1ff45985e847d698d0c93da28f4a764208c
Jul 29 16:51:24.843969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.843946    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s2gzd"
Jul 29 16:51:24.846863 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.846834    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c66261_984e_43ea_a1be_f63406b79d11.slice/crio-99b038d8633626f1caf298fc8c3c3de6c5aaf9750460c48345f1ff7beed79802 WatchSource:0}: Error finding container 99b038d8633626f1caf298fc8c3c3de6c5aaf9750460c48345f1ff7beed79802: Status 404 returned error can't find the container with id 99b038d8633626f1caf298fc8c3c3de6c5aaf9750460c48345f1ff7beed79802
Jul 29 16:51:24.851499 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.851475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvfl4"
Jul 29 16:51:24.853852 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.853822    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb01582e_77d0_49e4_9241_c3ab3a17cb59.slice/crio-9ebc3e2a7b6d8197381fe253e5fcc5bd559eefbb9ef317babffa4bfb97ed96e0 WatchSource:0}: Error finding container 9ebc3e2a7b6d8197381fe253e5fcc5bd559eefbb9ef317babffa4bfb97ed96e0: Status 404 returned error can't find the container with id 9ebc3e2a7b6d8197381fe253e5fcc5bd559eefbb9ef317babffa4bfb97ed96e0
Jul 29 16:51:24.860312 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.860270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:24.862397 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.862371    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cd7021_f3ac_471d_a272_c8ff4492fef5.slice/crio-c7603e25af86f0b97b8157fc9a365c7345774577594aaeb2595e85cfde067713 WatchSource:0}: Error finding container c7603e25af86f0b97b8157fc9a365c7345774577594aaeb2595e85cfde067713: Status 404 returned error can't find the container with id c7603e25af86f0b97b8157fc9a365c7345774577594aaeb2595e85cfde067713
Jul 29 16:51:24.867384 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:24.867361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:24.871289 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.871265    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4437f1a_045a_455c_9be1_92a486a48ab6.slice/crio-c555fc88438fc5c1db98ba31282169065f672d7e787c26f68ea373e55b9b7cb8 WatchSource:0}: Error finding container c555fc88438fc5c1db98ba31282169065f672d7e787c26f68ea373e55b9b7cb8: Status 404 returned error can't find the container with id c555fc88438fc5c1db98ba31282169065f672d7e787c26f68ea373e55b9b7cb8
Jul 29 16:51:24.877604 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:24.877579    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dbe303_c874_4a64_9bb0_b3da59e4ce8f.slice/crio-46b757cd61a5bcf23c94dd97ee3d7923782a59e2547cffdb32c5cc6cf99047e9 WatchSource:0}: Error finding container 46b757cd61a5bcf23c94dd97ee3d7923782a59e2547cffdb32c5cc6cf99047e9: Status 404 returned error can't find the container with id 46b757cd61a5bcf23c94dd97ee3d7923782a59e2547cffdb32c5cc6cf99047e9
Jul 29 16:51:25.001135 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.001058    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 16:51:25.116197 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.116150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:25.116370 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.116338    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:25.116435 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.116420    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:26.116401008 +0000 UTC m=+3.966434685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:25.217505 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.217455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:25.217693 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.217623    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:25.217778 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.217694    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:51:26.21767514 +0000 UTC m=+4.067708840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:25.318473 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.318380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:25.318644 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.318578    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:25.318644 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.318599    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:25.318644 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.318612    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:25.318832 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.318685    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:26.318664737 +0000 UTC m=+4.168698418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:25.552501 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.552376    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 16:46:24 +0000 UTC" deadline="2027-12-25 17:08:42.169569998 +0000 UTC"
Jul 29 16:51:25.552501 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.552438    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12336h17m16.617136536s"
Jul 29 16:51:25.699724 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.699553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:25.699873 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:25.699718    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:25.709603 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.709568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s2gzd" event={"ID":"eb01582e-77d0-49e4-9241-c3ab3a17cb59","Type":"ContainerStarted","Data":"9ebc3e2a7b6d8197381fe253e5fcc5bd559eefbb9ef317babffa4bfb97ed96e0"}
Jul 29 16:51:25.711017 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.710986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerStarted","Data":"9febbed8a89d5286462a62c5ab89b1ff45985e847d698d0c93da28f4a764208c"}
Jul 29 16:51:25.712923 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.712882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vc2dr" event={"ID":"5912cfcc-c880-49bd-90f2-86d4034e5d4d","Type":"ContainerStarted","Data":"65ba98a246a13d647cda00c845032de83ae6a93fa60e7b1b7968fb6ca106f066"}
Jul 29 16:51:25.714367 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.714341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jn7cv" event={"ID":"08906a19-3edf-49f2-8a85-6fbf739bbb2e","Type":"ContainerStarted","Data":"fb1ccbf26d90c8bec7dfc72ed03c6be950cd474073be352fb404deff2a6b0183"}
Jul 29 16:51:25.716280 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.716244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmbwc" event={"ID":"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f","Type":"ContainerStarted","Data":"46b757cd61a5bcf23c94dd97ee3d7923782a59e2547cffdb32c5cc6cf99047e9"}
Jul 29 16:51:25.717771 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.717729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"c555fc88438fc5c1db98ba31282169065f672d7e787c26f68ea373e55b9b7cb8"}
Jul 29 16:51:25.719557 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.719530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvfl4" event={"ID":"81cd7021-f3ac-471d-a272-c8ff4492fef5","Type":"ContainerStarted","Data":"c7603e25af86f0b97b8157fc9a365c7345774577594aaeb2595e85cfde067713"}
Jul 29 16:51:25.721755 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.721728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fk5zw" event={"ID":"63c66261-984e-43ea-a1be-f63406b79d11","Type":"ContainerStarted","Data":"99b038d8633626f1caf298fc8c3c3de6c5aaf9750460c48345f1ff7beed79802"}
Jul 29 16:51:25.723928 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:25.723875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2" event={"ID":"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d","Type":"ContainerStarted","Data":"1025884b4a86b13965fc2a51addcec51f50c993d822b257e1aca710a7777b8cc"}
Jul 29 16:51:26.126095 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.125412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:26.126095 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.125571    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:26.126095 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.125647    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:28.125626114 +0000 UTC m=+5.975659799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:26.226775 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.226185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:26.226775 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.226351    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:26.226775 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.226423    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:51:28.226403445 +0000 UTC m=+6.076437129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:26.327445 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.327405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:26.327644 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.327624    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:26.327713 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.327653    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:26.327713 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.327669    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:26.327812 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.327731    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:28.327711736 +0000 UTC m=+6.177745418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:26.553187 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.553149    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 16:46:24 +0000 UTC" deadline="2027-12-24 17:32:39.661518094 +0000 UTC"
Jul 29 16:51:26.553187 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.553184    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12312h41m13.108338377s"
Jul 29 16:51:26.700255 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.700229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:26.700468 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.700413    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:26.700933 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:26.700908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:26.701041 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:26.701021    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:27.699034 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:27.699000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:27.699635 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:27.699148    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:28.141492 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:28.141421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:28.141655 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.141546    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:28.141655 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.141591    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:32.141577941 +0000 UTC m=+9.991611620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:28.241830 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:28.241799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:28.242117 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.242098    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:28.242214 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.242168    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:51:32.242146799 +0000 UTC m=+10.092180491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:28.342374 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:28.342338    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:28.342584 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.342565    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:28.342662 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.342592    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:28.342662 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.342604    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:28.342764 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.342663    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:32.342643221 +0000 UTC m=+10.192676912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:28.698802 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:28.698771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:28.698981 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:28.698838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:28.698981 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.698967    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:28.699363 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:28.699335    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:29.699101 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:29.699052    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:29.699259 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:29.699201    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:30.699091 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:30.699041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:30.699533 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:30.699200    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:30.699533 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:30.699255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:30.699533 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:30.699352    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:31.698650 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:31.698618    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:31.698840 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:31.698758    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:32.177353 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:32.177258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:32.177791 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.177394    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:32.177791 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.177464    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:40.177444209 +0000 UTC m=+18.027477891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:32.277960 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:32.277893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:32.278135 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.278024    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:32.278135 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.278101    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:51:40.278069845 +0000 UTC m=+18.128103539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:32.379070 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:32.378983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:32.379244 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.379180    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:32.379244 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.379200    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:32.379244 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.379213    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:32.379402 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.379278    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:40.379260864 +0000 UTC m=+18.229294564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:32.700666 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:32.700072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:32.700666 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:32.700231    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:32.700666 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.700230    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:32.700666 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:32.700354    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:33.699229 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:33.699191    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:33.699680 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:33.699324    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:34.699061 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:34.699024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:34.699061 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:34.699053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:34.699299 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:34.699174    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:34.699609 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:34.699420    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:35.699206 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:35.699178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:35.699318 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:35.699279    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:36.698895 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:36.698857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:36.698895 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:36.698871    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:36.699148 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:36.698989    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:36.699148 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:36.699136    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:37.699423 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:37.699389    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:37.699868 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:37.699519    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:38.698764 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:38.698729    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:38.698959 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:38.698743    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:38.698959 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:38.698864    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:38.699056 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:38.698954    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:39.699097 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:39.699059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:39.699641 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:39.699207    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:40.237047 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:40.237004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:40.237267 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.237161    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:40.237267 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.237228    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:56.237209499 +0000 UTC m=+34.087243180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:40.337465 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:40.337424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:40.337647 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.337551    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:40.337647 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.337614    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:51:56.337599596 +0000 UTC m=+34.187633295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:40.438265 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:40.438227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:40.438460 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.438434    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:40.438524 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.438473    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:40.438524 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.438488    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:40.438618 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.438557    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:56.438538229 +0000 UTC m=+34.288571912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:40.698923 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:40.698833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:40.698923 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:40.698900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:40.699164 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.698966    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:40.699164 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:40.699017    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:41.699509 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:41.699474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:41.699984 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:41.699602    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:42.699638 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:42.699601    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:42.700109 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:42.699703    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:42.700109 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:42.699768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:42.700109 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:42.699837    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:43.699495 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:43.699357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:43.699594 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:43.699575    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:44.699472 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.699446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:44.700017 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.699490    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:44.700017 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:44.699574    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:44.700017 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:44.699689    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:44.759378 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.759346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s2gzd" event={"ID":"eb01582e-77d0-49e4-9241-c3ab3a17cb59","Type":"ContainerStarted","Data":"19dc081b865e353ad35b7d025fcd921b1fec194c6aab874dfc5d766057fe0ec8"}
Jul 29 16:51:44.761322 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.761293    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad7706df-e587-4d12-b151-37f895e4eeab" containerID="7e3f1ff206ff06d0edfcb098a196819a73288c11930b667d6618d16215b02931" exitCode=0
Jul 29 16:51:44.761435 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.761354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerDied","Data":"7e3f1ff206ff06d0edfcb098a196819a73288c11930b667d6618d16215b02931"}
Jul 29 16:51:44.763161 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.763057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vc2dr" event={"ID":"5912cfcc-c880-49bd-90f2-86d4034e5d4d","Type":"ContainerStarted","Data":"6f167735b3f8b713b7cd50fb07f842d0082704fcc9e4a75c6df4ea79c6899de8"}
Jul 29 16:51:44.764871 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.764850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jn7cv" event={"ID":"08906a19-3edf-49f2-8a85-6fbf739bbb2e","Type":"ContainerStarted","Data":"635c2123d28be6a9f6a083482cf3c92be4abb566c84fe97462c540872f3eca67"}
Jul 29 16:51:44.766706 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.766679    2559 generic.go:356] "Generic (PLEG): container finished" podID="6556c24d7e1dda4410307c41d995cf82" containerID="18530dda566f2df623eb062c1f48ee7341674f60f72bc4c47519518e020e4cde" exitCode=0
Jul 29 16:51:44.766830 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.766808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal" event={"ID":"6556c24d7e1dda4410307c41d995cf82","Type":"ContainerDied","Data":"18530dda566f2df623eb062c1f48ee7341674f60f72bc4c47519518e020e4cde"}
Jul 29 16:51:44.769398 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.769007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmbwc" event={"ID":"a9dbe303-c874-4a64-9bb0-b3da59e4ce8f","Type":"ContainerStarted","Data":"f31d535697d92506e5436c94a67a897ea82a90b617e31cb9daa543e1e0fa36a0"}
Jul 29 16:51:44.772827 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.772805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"82698a2fae0179f611b6df3540b17b377f8282c7cabbdc0665d21a7ff33b6eb3"}
Jul 29 16:51:44.772933 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.772884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"c8370de61fdb3832d540b466cb70e1c4fd07ffa3c55f76a4fdaaa32ba9566056"}
Jul 29 16:51:44.772991 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.772951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"39de4a5ff8584c2d0b991dbbecc2c25eaf1b28c9265037467eb25a6ff86cd3c9"}
Jul 29 16:51:44.773041 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.773001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"5b9f97828b933859b69e6ba062075d04208a78d3647fdc3600d26a2fb1067b31"}
Jul 29 16:51:44.773122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.773050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"c5dd42f5a637241adc32a9b826f3d29211aedebe9214cb93d81a9194c5e8d42a"}
Jul 29 16:51:44.773173 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.773159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"b85bd791567756766fe48cfc1a09e08299cbf393af1f91931b8eb349c4b5c6cd"}
Jul 29 16:51:44.774764 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.774598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvfl4" event={"ID":"81cd7021-f3ac-471d-a272-c8ff4492fef5","Type":"ContainerStarted","Data":"d143c7550a8d47cfe1aaf70433ad81a0942e1e07a1a4b811b846fcc76afd8966"}
Jul 29 16:51:44.776243 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.776222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2" event={"ID":"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d","Type":"ContainerStarted","Data":"f5ae5aca9a46c1330367c328ed4179e637e82b29af04902f4b2b5e726c861ec4"}
Jul 29 16:51:44.778029 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.778005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal" event={"ID":"bfca4edb4041480c1eb9593a15543d37","Type":"ContainerStarted","Data":"2d788c6e98f31cd87a0aea3995c88c1e9bb47eeff70e81b8ea7d7c1dc80d2f4e"}
Jul 29 16:51:44.780862 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.780798    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s2gzd" podStartSLOduration=2.987967331 podStartE2EDuration="21.78078169s" totalImagesPullingTime="18.792814359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:44.780283422 +0000 UTC m=+22.630317121" watchObservedRunningTime="2026-07-29 16:51:44.78078169 +0000 UTC m=+22.630815389"
Jul 29 16:51:44.794671 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.794630    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-76.ec2.internal" podStartSLOduration=21.794617876 podStartE2EDuration="21.794617876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:44.7944201 +0000 UTC m=+22.644453800" watchObservedRunningTime="2026-07-29 16:51:44.794617876 +0000 UTC m=+22.644651577"
Jul 29 16:51:44.810154 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.810111    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jn7cv" podStartSLOduration=4.051502274 podStartE2EDuration="22.810094917s" totalImagesPullingTime="18.758592643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:44.809624818 +0000 UTC m=+22.659658517" watchObservedRunningTime="2026-07-29 16:51:44.810094917 +0000 UTC m=+22.660128617"
Jul 29 16:51:44.831140 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.831102    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vc2dr" podStartSLOduration=2.9901835930000003 podStartE2EDuration="21.831068327s" totalImagesPullingTime="18.840884734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:44.830365261 +0000 UTC m=+22.680398959" watchObservedRunningTime="2026-07-29 16:51:44.831068327 +0000 UTC m=+22.681102026"
Jul 29 16:51:44.845238 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.844999    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lvfl4" podStartSLOduration=3.06378642 podStartE2EDuration="21.844982728s" totalImagesPullingTime="18.781196308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:44.844863487 +0000 UTC m=+22.694897187" watchObservedRunningTime="2026-07-29 16:51:44.844982728 +0000 UTC m=+22.695016427"
Jul 29 16:51:44.876990 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:44.876951    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dmbwc" podStartSLOduration=3.112037689 podStartE2EDuration="21.87693364s" totalImagesPullingTime="18.764895951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:44.860674453 +0000 UTC m=+22.710708151" watchObservedRunningTime="2026-07-29 16:51:44.87693364 +0000 UTC m=+22.726967340"
Jul 29 16:51:45.638946 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.638923    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 16:51:45.698995 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.698956    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:45.699183 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:45.699077    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:45.783091 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.782993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal" event={"ID":"6556c24d7e1dda4410307c41d995cf82","Type":"ContainerStarted","Data":"37c7dc73d2ed7163be25eefe307503a78a4b92bae8617cb8a5f1cbef9d3a97dd"}
Jul 29 16:51:45.784681 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.784651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fk5zw" event={"ID":"63c66261-984e-43ea-a1be-f63406b79d11","Type":"ContainerStarted","Data":"5d828c53661373ae815582cc5dfd3e446639e851c0d54f1b4035e796484cba29"}
Jul 29 16:51:45.786745 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.786708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2" event={"ID":"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d","Type":"ContainerStarted","Data":"7ef06578174c7bfe84110630457a56d983833a58a8ae6bcb08e9384c06289bcf"}
Jul 29 16:51:45.801524 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.801479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-76.ec2.internal" podStartSLOduration=22.801464707 podStartE2EDuration="22.801464707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:45.801013256 +0000 UTC m=+23.651046954" watchObservedRunningTime="2026-07-29 16:51:45.801464707 +0000 UTC m=+23.651498408"
Jul 29 16:51:45.818882 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:45.818837    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fk5zw" podStartSLOduration=4.093237409 podStartE2EDuration="22.818822436s" totalImagesPullingTime="18.725585027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:45.818788185 +0000 UTC m=+23.668821884" watchObservedRunningTime="2026-07-29 16:51:45.818822436 +0000 UTC m=+23.668856134"
Jul 29 16:51:46.335417 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.335382    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:46.336067 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.336044    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:46.569783 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.569669    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T16:51:45.638942189Z","UUID":"38e88217-5159-4cde-bc34-c0e02933e1c2","Handler":null,"Name":"","Endpoint":""}
Jul 29 16:51:46.571422 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.571396    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 16:51:46.571563 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.571439    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 16:51:46.698796 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.698684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:46.698944 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:46.698822    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:46.698944 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.698846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:46.699023 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:46.698997    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:46.805076 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.805053    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:46.805846 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:46.805826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dmbwc"
Jul 29 16:51:47.676630 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:47.676581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:47.698530 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:47.698506    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:47.698662 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:47.698613    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:47.795968 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:47.795932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"742d07b53a89dec216065186c0003ca692af5cdc45a69ae3bef5ba6d4be7cda9"}
Jul 29 16:51:47.798177 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:47.798118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2" event={"ID":"5be0b3c8-8dd8-4ef3-b4d1-b25fd3647e5d","Type":"ContainerStarted","Data":"3ec3cfcba89def89933050cda494d2d14b2cada5c413a6b55e452b7cceefa3f7"}
Jul 29 16:51:47.818271 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:47.818230    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdss2" podStartSLOduration=3.519929967 podStartE2EDuration="24.818214144s" totalImagesPullingTime="21.298284177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:47.817719466 +0000 UTC m=+25.667753175" watchObservedRunningTime="2026-07-29 16:51:47.818214144 +0000 UTC m=+25.668247846"
Jul 29 16:51:48.699097 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:48.699052    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:48.699236 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:48.699107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:48.699236 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:48.699201    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:48.699326 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:48.699308    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:48.729015 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:48.728987    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:48.829788 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:48.829719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:49.698998 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.698795    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:49.699136 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:49.699021    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:49.803486 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.803453    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad7706df-e587-4d12-b151-37f895e4eeab" containerID="a21ea5f4298f11eaae806689428f34902d1b9427a097d07c75fc537f9b58c837" exitCode=0
Jul 29 16:51:49.803602 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.803537    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerDied","Data":"a21ea5f4298f11eaae806689428f34902d1b9427a097d07c75fc537f9b58c837"}
Jul 29 16:51:49.807286 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.807260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" event={"ID":"b4437f1a-045a-455c-9be1-92a486a48ab6","Type":"ContainerStarted","Data":"f531676ca046f752f608723b18414f480966d008694a51b77b5e5802907cc466"}
Jul 29 16:51:49.807595 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.807580    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:49.807660 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.807634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:49.822090 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.822067    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:49.856636 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:49.856598    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5" podStartSLOduration=7.623695744 podStartE2EDuration="26.856580401s" totalImagesPullingTime="19.232884657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:49.856164808 +0000 UTC m=+27.706198502" watchObservedRunningTime="2026-07-29 16:51:49.856580401 +0000 UTC m=+27.706614124"
Jul 29 16:51:50.352879 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.352815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:50.453346 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.453305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:50.698898 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.698831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:50.699052 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.698831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:50.699052 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:50.698932    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:50.699052 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:50.699031    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:50.811356 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.811317    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad7706df-e587-4d12-b151-37f895e4eeab" containerID="3d644ff313bffe060dcc5e76f2cc13d6d39abc57d30e47265dfcadfeda70ffbc" exitCode=0
Jul 29 16:51:50.811356 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.811351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerDied","Data":"3d644ff313bffe060dcc5e76f2cc13d6d39abc57d30e47265dfcadfeda70ffbc"}
Jul 29 16:51:50.811980 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.811961    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:50.827221 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:50.827197    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:51:51.139618 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.139320    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gkqgp"]
Jul 29 16:51:51.140182 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.139663    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:51.140182 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:51.139776    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:51.145346 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.145316    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ntfhp"]
Jul 29 16:51:51.145484 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.145410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:51.145547 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:51.145504    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:51.146245 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.146215    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stljd"]
Jul 29 16:51:51.146325 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.146306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:51.146397 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:51.146381    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:51.403036 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.402956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:51.474403 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:51.474370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:52.216746 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.216710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:52.317712 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.317666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:52.592768 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:52.592739    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7706df_e587_4d12_b151_37f895e4eeab.slice/crio-conmon-8e2b6eb1e9c1fa7b24c51eca43a4666019875ed59b9f171b3d00f51acfcb0856.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 16:51:52.700014 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.699982    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:52.700198 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:52.700062    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:52.700198 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.700109    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:52.700303 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:52.700196    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:52.700303 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.700220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:52.700303 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:52.700277    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:52.818505 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.818428    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad7706df-e587-4d12-b151-37f895e4eeab" containerID="8e2b6eb1e9c1fa7b24c51eca43a4666019875ed59b9f171b3d00f51acfcb0856" exitCode=0
Jul 29 16:51:52.818639 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:52.818512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerDied","Data":"8e2b6eb1e9c1fa7b24c51eca43a4666019875ed59b9f171b3d00f51acfcb0856"}
Jul 29 16:51:53.579508 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:53.579421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:53.680393 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:53.680353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:54.578368 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:54.578325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:54.639786 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:54.639743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:54.699352 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:54.699324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:54.699519 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:54.699328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:54.699519 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:54.699459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:54.699519 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:54.699453    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:54.699692 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:54.699563    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:54.699692 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:54.699664    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:55.543518 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:55.543477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:56.261018 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.260975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:56.261478 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.261164    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:56.261478 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.261237    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs podName:f1c52131-cd16-415a-a195-0d47aca1a9c4 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:28.261219444 +0000 UTC m=+66.111253140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs") pod "network-metrics-daemon-stljd" (UID: "f1c52131-cd16-415a-a195-0d47aca1a9c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 16:51:56.362247 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.362208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:56.362412 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.362345    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:56.362412 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.362404    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret podName:f51ad380-8ca6-47ae-be18-66d23531f26f nodeName:}" failed. No retries permitted until 2026-07-29 16:52:28.362389886 +0000 UTC m=+66.212423562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret") pod "global-pull-secret-syncer-gkqgp" (UID: "f51ad380-8ca6-47ae-be18-66d23531f26f") : object "kube-system"/"original-pull-secret" not registered
Jul 29 16:51:56.463024 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.462993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:56.463227 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.463189    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 16:51:56.463227 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.463211    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 16:51:56.463227 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.463220    2559 projected.go:196] Error preparing data for projected volume kube-api-access-6jlnx for pod openshift-network-diagnostics/network-check-target-ntfhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:56.463384 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.463268    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx podName:a1c31620-e5ed-42b4-a0b1-e6b412624dc6 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:28.463254759 +0000 UTC m=+66.313288435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jlnx" (UniqueName: "kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx") pod "network-check-target-ntfhp" (UID: "a1c31620-e5ed-42b4-a0b1-e6b412624dc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 16:51:56.535132 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.534966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:56.602691 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.602615    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:56.698692 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.698654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:56.698872 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.698658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:56.698872 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.698778    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkqgp" podUID="f51ad380-8ca6-47ae-be18-66d23531f26f"
Jul 29 16:51:56.698872 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:56.698809    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:56.699030 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.698904    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stljd" podUID="f1c52131-cd16-415a-a195-0d47aca1a9c4"
Jul 29 16:51:56.699030 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:56.698986    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfhp" podUID="a1c31620-e5ed-42b4-a0b1-e6b412624dc6"
Jul 29 16:51:57.832309 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:57.832132    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad7706df-e587-4d12-b151-37f895e4eeab" containerID="15059541da7fef5d0dc0a0ac094308f59e9e426c112498fc308d077c75439e8f" exitCode=0
Jul 29 16:51:57.832738 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:57.832212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerDied","Data":"15059541da7fef5d0dc0a0ac094308f59e9e426c112498fc308d077c75439e8f"}
Jul 29 16:51:58.015464 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.015432    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-76.ec2.internal" event="NodeReady"
Jul 29 16:51:58.015577 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.015567    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 16:51:58.051953 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.051926    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b54f7fb5-hvl2p"]
Jul 29 16:51:58.063258 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.063237    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.064855 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.064828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.066233 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.066217    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 16:51:58.066233 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.066233    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-78qw7\""
Jul 29 16:51:58.066468 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.066455    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 16:51:58.066623 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.066606    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 16:51:58.068836 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.068815    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tbq6b"]
Jul 29 16:51:58.073427 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.073407    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 16:51:58.079255 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.079239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b54f7fb5-hvl2p"]
Jul 29 16:51:58.079326 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.079265    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q9f27"]
Jul 29 16:51:58.079363 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.079340    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.082121 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.082105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 16:51:58.082215 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.082120    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2hb68\""
Jul 29 16:51:58.082215 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.082153    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 16:51:58.093331 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.093313    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tbq6b"]
Jul 29 16:51:58.093415 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.093345    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q9f27"]
Jul 29 16:51:58.093415 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.093403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.096001 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.095983    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dg98s\""
Jul 29 16:51:58.096131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.096033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 16:51:58.096131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.096043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 16:51:58.096131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.096053    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 16:51:58.129259 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.129234    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.174268 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-trusted-ca\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174268 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174268    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-bound-sa-token\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbv4\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-kube-api-access-mcbv4\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kn6\" (UniqueName: \"kubernetes.io/projected/500cedb2-76db-4404-88b6-21e82828a29c-kube-api-access-d5kn6\") pod \"ingress-canary-q9f27\" (UID: \"500cedb2-76db-4404-88b6-21e82828a29c\") " pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edb857fd-9890-4518-818c-af126e650336-ca-trust-extracted\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174385    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-installation-pull-secrets\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174413    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-tmp-dir\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.174444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsc8\" (UniqueName: \"kubernetes.io/projected/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-kube-api-access-6dsc8\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.174662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-registry-certificates\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.174662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-config-volume\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.174662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/500cedb2-76db-4404-88b6-21e82828a29c-cert\") pod \"ingress-canary-q9f27\" (UID: \"500cedb2-76db-4404-88b6-21e82828a29c\") " pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.174662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.174543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-image-registry-private-configuration\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.233928 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.233891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.255005 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.254978    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6n5vf"]
Jul 29 16:51:58.275768 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-image-registry-private-configuration\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.275866 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-trusted-ca\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.275866 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-bound-sa-token\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.275866 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbv4\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-kube-api-access-mcbv4\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.275866 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.276033 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kn6\" (UniqueName: \"kubernetes.io/projected/500cedb2-76db-4404-88b6-21e82828a29c-kube-api-access-d5kn6\") pod \"ingress-canary-q9f27\" (UID: \"500cedb2-76db-4404-88b6-21e82828a29c\") " pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.276033 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edb857fd-9890-4518-818c-af126e650336-ca-trust-extracted\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.276033 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.275944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-installation-pull-secrets\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.276033 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.276014    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:58.276262 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-tmp-dir\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.276262 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.276097    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls podName:bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c nodeName:}" failed. No retries permitted until 2026-07-29 16:51:58.776058729 +0000 UTC m=+36.626092413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls") pod "dns-default-tbq6b" (UID: "bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:58.276262 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsc8\" (UniqueName: \"kubernetes.io/projected/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-kube-api-access-6dsc8\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.276262 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.276262 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-registry-certificates\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.276262 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-config-volume\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.276538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/500cedb2-76db-4404-88b6-21e82828a29c-cert\") pod \"ingress-canary-q9f27\" (UID: \"500cedb2-76db-4404-88b6-21e82828a29c\") " pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.276538 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.276301    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:58.276538 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.276317    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b54f7fb5-hvl2p: secret "image-registry-tls" not found
Jul 29 16:51:58.276538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edb857fd-9890-4518-818c-af126e650336-ca-trust-extracted\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.276538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-tmp-dir\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.276538 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.276362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls podName:edb857fd-9890-4518-818c-af126e650336 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:58.776345553 +0000 UTC m=+36.626379241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls") pod "image-registry-57b54f7fb5-hvl2p" (UID: "edb857fd-9890-4518-818c-af126e650336") : secret "image-registry-tls" not found
Jul 29 16:51:58.276828 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-registry-certificates\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.276895 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.276862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-config-volume\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.277208 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.277187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-trusted-ca\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.280000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.279977    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"]
Jul 29 16:51:58.280163 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.280133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.280676 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.280630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/500cedb2-76db-4404-88b6-21e82828a29c-cert\") pod \"ingress-canary-q9f27\" (UID: \"500cedb2-76db-4404-88b6-21e82828a29c\") " pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.281249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.280939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-installation-pull-secrets\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.281249 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.281211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-image-registry-private-configuration\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.284762 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.283558    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbp95\""
Jul 29 16:51:58.284762 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.283935    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 16:51:58.284762 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.284386    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 16:51:58.284762 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.284591    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 16:51:58.285008 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.284767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 16:51:58.293223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.293201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbv4\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-kube-api-access-mcbv4\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.293723 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.293706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kn6\" (UniqueName: \"kubernetes.io/projected/500cedb2-76db-4404-88b6-21e82828a29c-kube-api-access-d5kn6\") pod \"ingress-canary-q9f27\" (UID: \"500cedb2-76db-4404-88b6-21e82828a29c\") " pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.294548 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.294527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-bound-sa-token\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.304288 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.304269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsc8\" (UniqueName: \"kubernetes.io/projected/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-kube-api-access-6dsc8\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.314415 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.314396    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6n5vf"]
Jul 29 16:51:58.314487 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.314441    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"]
Jul 29 16:51:58.314532 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.314508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"
Jul 29 16:51:58.318326 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.318303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 16:51:58.318412 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.318394    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tvq5v\""
Jul 29 16:51:58.318711 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.318696    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 16:51:58.333274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.333252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.377035 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.376972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c061c174-6b97-4f67-b636-6831a6862ace-data-volume\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.377035 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.377001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.377189 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.377076    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.377189 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.377156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c061c174-6b97-4f67-b636-6831a6862ace-crio-socket\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.377252 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.377204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bvr\" (UniqueName: \"kubernetes.io/projected/c061c174-6b97-4f67-b636-6831a6862ace-kube-api-access-89bvr\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.401204 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.401184    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q9f27"
Jul 29 16:51:58.433583 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.433563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.477664 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c061c174-6b97-4f67-b636-6831a6862ace-data-volume\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.477664 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.477881 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7cb\" (UniqueName: \"kubernetes.io/projected/458ca5ca-78b2-498f-93df-43015cf80a74-kube-api-access-jd7cb\") pod \"migrator-6865c9967-dwbc2\" (UID: \"458ca5ca-78b2-498f-93df-43015cf80a74\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"
Jul 29 16:51:58.477881 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.477765    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.477881 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.477881 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.477834    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:51:58.977819062 +0000 UTC m=+36.827852765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.477881 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c061c174-6b97-4f67-b636-6831a6862ace-crio-socket\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.478180 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.477889    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.478180 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bvr\" (UniqueName: \"kubernetes.io/projected/c061c174-6b97-4f67-b636-6831a6862ace-kube-api-access-89bvr\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.478180 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.477941    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:51:58.977926286 +0000 UTC m=+36.827959978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.478180 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.477990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c061c174-6b97-4f67-b636-6831a6862ace-crio-socket\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.478180 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.478028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c061c174-6b97-4f67-b636-6831a6862ace-data-volume\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.488856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.488835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bvr\" (UniqueName: \"kubernetes.io/projected/c061c174-6b97-4f67-b636-6831a6862ace-kube-api-access-89bvr\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.535920 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.535883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.578621 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.578591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7cb\" (UniqueName: \"kubernetes.io/projected/458ca5ca-78b2-498f-93df-43015cf80a74-kube-api-access-jd7cb\") pod \"migrator-6865c9967-dwbc2\" (UID: \"458ca5ca-78b2-498f-93df-43015cf80a74\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"
Jul 29 16:51:58.584887 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.584862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q9f27"]
Jul 29 16:51:58.588305 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:58.588282    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500cedb2_76db_4404_88b6_21e82828a29c.slice/crio-d474ca446302d7e848e5ce4b9941cc9c578d2f762bdbc01f3b32bd230dcd934c WatchSource:0}: Error finding container d474ca446302d7e848e5ce4b9941cc9c578d2f762bdbc01f3b32bd230dcd934c: Status 404 returned error can't find the container with id d474ca446302d7e848e5ce4b9941cc9c578d2f762bdbc01f3b32bd230dcd934c
Jul 29 16:51:58.588953 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.588931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7cb\" (UniqueName: \"kubernetes.io/projected/458ca5ca-78b2-498f-93df-43015cf80a74-kube-api-access-jd7cb\") pod \"migrator-6865c9967-dwbc2\" (UID: \"458ca5ca-78b2-498f-93df-43015cf80a74\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"
Jul 29 16:51:58.622466 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.622443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"
Jul 29 16:51:58.699172 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.699147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:51:58.699290 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.699148    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:51:58.699352 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.699159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:51:58.704567 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.704549    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 16:51:58.704767 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.704683    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 16:51:58.704767 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.704720    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fgvc\""
Jul 29 16:51:58.704856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.704808    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qtk2\""
Jul 29 16:51:58.705012 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.704996    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 16:51:58.705522 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.705508    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 16:51:58.765009 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.764985    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2"]
Jul 29 16:51:58.767423 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:51:58.767401    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458ca5ca_78b2_498f_93df_43015cf80a74.slice/crio-0409812078ae7562df9c826bf929187edffe51bcddf2dc90f19e7fc961aefc3d WatchSource:0}: Error finding container 0409812078ae7562df9c826bf929187edffe51bcddf2dc90f19e7fc961aefc3d: Status 404 returned error can't find the container with id 0409812078ae7562df9c826bf929187edffe51bcddf2dc90f19e7fc961aefc3d
Jul 29 16:51:58.780164 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.780144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:58.780257 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.780214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:58.780316 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.780273    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:58.780354 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.780318    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls podName:bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c nodeName:}" failed. No retries permitted until 2026-07-29 16:51:59.780304975 +0000 UTC m=+37.630338651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls") pod "dns-default-tbq6b" (UID: "bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:58.780354 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.780348    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:58.780354 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.780364    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b54f7fb5-hvl2p: secret "image-registry-tls" not found
Jul 29 16:51:58.780507 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.780410    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls podName:edb857fd-9890-4518-818c-af126e650336 nodeName:}" failed. No retries permitted until 2026-07-29 16:51:59.780394413 +0000 UTC m=+37.630428090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls") pod "image-registry-57b54f7fb5-hvl2p" (UID: "edb857fd-9890-4518-818c-af126e650336") : secret "image-registry-tls" not found
Jul 29 16:51:58.790643 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.790616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.835379 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.835357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2" event={"ID":"458ca5ca-78b2-498f-93df-43015cf80a74","Type":"ContainerStarted","Data":"0409812078ae7562df9c826bf929187edffe51bcddf2dc90f19e7fc961aefc3d"}
Jul 29 16:51:58.838301 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.838277    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad7706df-e587-4d12-b151-37f895e4eeab" containerID="939849246cbb485e3fa01d63aa4dfa7713d2da484066b88f1d5d255a8c6705f1" exitCode=0
Jul 29 16:51:58.838381 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.838319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerDied","Data":"939849246cbb485e3fa01d63aa4dfa7713d2da484066b88f1d5d255a8c6705f1"}
Jul 29 16:51:58.839444 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.839422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q9f27" event={"ID":"500cedb2-76db-4404-88b6-21e82828a29c","Type":"ContainerStarted","Data":"d474ca446302d7e848e5ce4b9941cc9c578d2f762bdbc01f3b32bd230dcd934c"}
Jul 29 16:51:58.877056 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.877032    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:51:58.982415 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.982222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.982525 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.982372    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.982571 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.982548    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:51:59.982532587 +0000 UTC m=+37.832566265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:58.982629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:58.982619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:58.982719 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.982704    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:51:58.982799 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:58.982767    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:51:59.982751897 +0000 UTC m=+37.832785574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : configmap "kube-rbac-proxy" not found
Jul 29 16:51:59.790017 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.789984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:51:59.790162 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.790092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:51:59.790221 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.790165    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:51:59.790221 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.790187    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b54f7fb5-hvl2p: secret "image-registry-tls" not found
Jul 29 16:51:59.790221 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.790208    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:51:59.790351 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.790259    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls podName:edb857fd-9890-4518-818c-af126e650336 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:01.790238054 +0000 UTC m=+39.640271741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls") pod "image-registry-57b54f7fb5-hvl2p" (UID: "edb857fd-9890-4518-818c-af126e650336") : secret "image-registry-tls" not found
Jul 29 16:51:59.790351 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.790280    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls podName:bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c nodeName:}" failed. No retries permitted until 2026-07-29 16:52:01.790270589 +0000 UTC m=+39.640304269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls") pod "dns-default-tbq6b" (UID: "bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c") : secret "dns-default-metrics-tls" not found
Jul 29 16:51:59.834883 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.834853    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jn7cv_08906a19-3edf-49f2-8a85-6fbf739bbb2e/dns-node-resolver/0.log"
Jul 29 16:51:59.846414 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.846387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" event={"ID":"ad7706df-e587-4d12-b151-37f895e4eeab","Type":"ContainerStarted","Data":"15644f15716077b5984747a107824291d5197e8eb0436ff2060ffe9806bb28a1"}
Jul 29 16:51:59.873984 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.873943    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4w4tf" podStartSLOduration=7.006609454 podStartE2EDuration="36.873928855s" totalImagesPullingTime="29.867319401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:51:59.87206039 +0000 UTC m=+37.722094101" watchObservedRunningTime="2026-07-29 16:51:59.873928855 +0000 UTC m=+37.723962550"
Jul 29 16:51:59.991859 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.991826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:59.992028 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:51:59.991924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:51:59.992028 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.991992    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:51:59.992161 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.992064    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:52:01.992043171 +0000 UTC m=+39.842076865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : configmap "kube-rbac-proxy" not found
Jul 29 16:51:59.992377 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.992359    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:51:59.992447 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:51:59.992424    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:52:01.992407597 +0000 UTC m=+39.842441289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:00.001889 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.001844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:00.105001 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.104921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:00.372365 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.372288    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2zcp8"]
Jul 29 16:52:00.389844 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.389818    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2zcp8"]
Jul 29 16:52:00.390014 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.389920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.392826 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.392781    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 16:52:00.392967 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.392860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pbv78\""
Jul 29 16:52:00.392967 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.392911    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 16:52:00.393074 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.392982    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 16:52:00.394053 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.394033    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 16:52:00.414231 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.414206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:00.495651 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.495623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bffc583f-d761-4581-b4d5-58464e1d51be-signing-cabundle\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.495779 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.495660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwr7j\" (UniqueName: \"kubernetes.io/projected/bffc583f-d761-4581-b4d5-58464e1d51be-kube-api-access-mwr7j\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.495779 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.495712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bffc583f-d761-4581-b4d5-58464e1d51be-signing-key\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.596700 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.596657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bffc583f-d761-4581-b4d5-58464e1d51be-signing-cabundle\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.596864 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.596800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwr7j\" (UniqueName: \"kubernetes.io/projected/bffc583f-d761-4581-b4d5-58464e1d51be-kube-api-access-mwr7j\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.596864 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.596859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bffc583f-d761-4581-b4d5-58464e1d51be-signing-key\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.597458 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.597435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bffc583f-d761-4581-b4d5-58464e1d51be-signing-cabundle\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.601257 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.601233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bffc583f-d761-4581-b4d5-58464e1d51be-signing-key\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.607647 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.607629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwr7j\" (UniqueName: \"kubernetes.io/projected/bffc583f-d761-4581-b4d5-58464e1d51be-kube-api-access-mwr7j\") pod \"service-ca-658d7b7c78-2zcp8\" (UID: \"bffc583f-d761-4581-b4d5-58464e1d51be\") " pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.699798 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.699777    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8"
Jul 29 16:52:00.839421 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:00.839276    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2zcp8"]
Jul 29 16:52:00.880981 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:00.880954    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffc583f_d761_4581_b4d5_58464e1d51be.slice/crio-4748e4e744df997740baaaef8dbe6866acb7d541bab149df248d0ca97f62ea79 WatchSource:0}: Error finding container 4748e4e744df997740baaaef8dbe6866acb7d541bab149df248d0ca97f62ea79: Status 404 returned error can't find the container with id 4748e4e744df997740baaaef8dbe6866acb7d541bab149df248d0ca97f62ea79
Jul 29 16:52:01.041320 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:01.041290    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvfl4_81cd7021-f3ac-471d-a272-c8ff4492fef5/node-ca/0.log"
Jul 29 16:52:01.808336 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:01.808295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:01.808504 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:01.808423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:01.808504 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:01.808465    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:52:01.808504 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:01.808484    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b54f7fb5-hvl2p: secret "image-registry-tls" not found
Jul 29 16:52:01.808620 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:01.808550    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls podName:edb857fd-9890-4518-818c-af126e650336 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:05.808529669 +0000 UTC m=+43.658563352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls") pod "image-registry-57b54f7fb5-hvl2p" (UID: "edb857fd-9890-4518-818c-af126e650336") : secret "image-registry-tls" not found
Jul 29 16:52:01.808620 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:01.808552    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:52:01.808620 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:01.808617    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls podName:bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c nodeName:}" failed. No retries permitted until 2026-07-29 16:52:05.808601621 +0000 UTC m=+43.658635301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls") pod "dns-default-tbq6b" (UID: "bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c") : secret "dns-default-metrics-tls" not found
Jul 29 16:52:01.853396 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:01.853357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8" event={"ID":"bffc583f-d761-4581-b4d5-58464e1d51be","Type":"ContainerStarted","Data":"4748e4e744df997740baaaef8dbe6866acb7d541bab149df248d0ca97f62ea79"}
Jul 29 16:52:01.854884 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:01.854843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q9f27" event={"ID":"500cedb2-76db-4404-88b6-21e82828a29c","Type":"ContainerStarted","Data":"0681f16aa1543e98c313dfcad3b020089cf4a4599ab6b66c857730d0a9f1f170"}
Jul 29 16:52:01.885791 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:01.885743    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q9f27" podStartSLOduration=1.577931629 podStartE2EDuration="3.885725136s" totalImagesPullingTime="2.307793507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:01.884948477 +0000 UTC m=+39.734982179" watchObservedRunningTime="2026-07-29 16:52:01.885725136 +0000 UTC m=+39.735758837"
Jul 29 16:52:02.009936 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.009904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:02.010144 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.010004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:02.010144 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:02.010041    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 16:52:02.010144 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:02.010138    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:52:06.010116621 +0000 UTC m=+43.860150308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : configmap "kube-rbac-proxy" not found
Jul 29 16:52:02.010313 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:02.010159    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:02.010313 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:02.010216    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:52:06.010200978 +0000 UTC m=+43.860234654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:02.464113 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.464017    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:02.547928 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.547892    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:02.764094 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.764052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:02.859067 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.859036    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2" event={"ID":"458ca5ca-78b2-498f-93df-43015cf80a74","Type":"ContainerStarted","Data":"93448f314e086978ba986a7204c4a7ac4180ee04f41a93ea24edc85e580c65bc"}
Jul 29 16:52:02.859184 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.859122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2" event={"ID":"458ca5ca-78b2-498f-93df-43015cf80a74","Type":"ContainerStarted","Data":"61444e55ee8cf785c1f3cb061bbde718b423447351b3e10691995867d01280e7"}
Jul 29 16:52:02.880626 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.880586    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dwbc2" podStartSLOduration=1.019240234 podStartE2EDuration="4.880565546s" totalImagesPullingTime="3.861325312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:02.880169249 +0000 UTC m=+40.730202947" watchObservedRunningTime="2026-07-29 16:52:02.880565546 +0000 UTC m=+40.730599245"
Jul 29 16:52:02.943714 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:02.943684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:03.862646 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:03.862607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8" event={"ID":"bffc583f-d761-4581-b4d5-58464e1d51be","Type":"ContainerStarted","Data":"c2a2cb08ca92e64f760490c8ac16c4147aabdd10e766d473ce29d4a75dab593f"}
Jul 29 16:52:03.881575 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:03.881529    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-2zcp8" podStartSLOduration=1.7494833459999999 podStartE2EDuration="3.881516043s" totalImagesPullingTime="2.132032697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:03.881006546 +0000 UTC m=+41.731040244" watchObservedRunningTime="2026-07-29 16:52:03.881516043 +0000 UTC m=+41.731549777"
Jul 29 16:52:05.269509 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:05.269467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:05.369315 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:05.369279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:05.734940 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:05.734847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:05.840178 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:05.840146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:05.840368 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:05.840205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:05.840368 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:05.840305    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 16:52:05.840368 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:05.840318    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b54f7fb5-hvl2p: secret "image-registry-tls" not found
Jul 29 16:52:05.840368 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:05.840318    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 16:52:05.840517 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:05.840375    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls podName:edb857fd-9890-4518-818c-af126e650336 nodeName:}" failed. No retries permitted until 2026-07-29 16:52:13.840354956 +0000 UTC m=+51.690388632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls") pod "image-registry-57b54f7fb5-hvl2p" (UID: "edb857fd-9890-4518-818c-af126e650336") : secret "image-registry-tls" not found
Jul 29 16:52:05.840517 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:05.840388    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls podName:bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c nodeName:}" failed. No retries permitted until 2026-07-29 16:52:13.84038247 +0000 UTC m=+51.690416146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls") pod "dns-default-tbq6b" (UID: "bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c") : secret "dns-default-metrics-tls" not found
Jul 29 16:52:06.041951 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:06.041865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:06.041951 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:06.041928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:06.042150 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:06.042032    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:06.042150 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:06.042132    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls podName:c061c174-6b97-4f67-b636-6831a6862ace nodeName:}" failed. No retries permitted until 2026-07-29 16:52:14.042110792 +0000 UTC m=+51.892144473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n5vf" (UID: "c061c174-6b97-4f67-b636-6831a6862ace") : secret "insights-runtime-extractor-tls" not found
Jul 29 16:52:06.042563 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:06.042544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c061c174-6b97-4f67-b636-6831a6862ace-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:06.204052 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:06.204016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:06.404346 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:06.404256    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:06.819882 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:06.819804    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:07.007380 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:07.007327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:07.430447 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:07.430418    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.204952 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.204917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.302545 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.302516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.404074 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.404041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.527230 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.527147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.601574 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.601548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.704655 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.704623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.829830 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.829744    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:08.950246 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:08.950215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:09.203750 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:09.203718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:09.965667 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:09.965634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:10.765156 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:10.765118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:10.862227 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:10.862199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:11.025956 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:11.025891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:11.749320 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:11.749283    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:11.849818 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:11.849782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:12.012653 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:12.012561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:12.404039 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:12.403933    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:12.609128 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:12.609096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:13.203131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.203103    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:13.365938 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.365906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:13.466746 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.466672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:13.601853 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.601820    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:13.852445 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.852349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:13.896710 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.896678    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:13.896871 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.896749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:13.899865 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.899844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c-metrics-tls\") pod \"dns-default-tbq6b\" (UID: \"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c\") " pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:13.899966 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.899945    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"image-registry-57b54f7fb5-hvl2p\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") " pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:13.979449 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.979421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:13.988398 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:13.988375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:14.004879 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.004848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:14.099413 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.099376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:14.101856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.101832    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c061c174-6b97-4f67-b636-6831a6862ace-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n5vf\" (UID: \"c061c174-6b97-4f67-b636-6831a6862ace\") " pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:14.116487 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.116464    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b54f7fb5-hvl2p"]
Jul 29 16:52:14.131331 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.131305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tbq6b"]
Jul 29 16:52:14.134144 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:14.134118    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb25ba8_1b1b_40cd_9a7a_301beaeecd5c.slice/crio-33b6c260709e283328676edc011d303b751867bdf0117530488314d5b16f1086 WatchSource:0}: Error finding container 33b6c260709e283328676edc011d303b751867bdf0117530488314d5b16f1086: Status 404 returned error can't find the container with id 33b6c260709e283328676edc011d303b751867bdf0117530488314d5b16f1086
Jul 29 16:52:14.206206 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.206179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6n5vf"
Jul 29 16:52:14.345969 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.345936    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6n5vf"]
Jul 29 16:52:14.348926 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:14.348891    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc061c174_6b97_4f67_b636_6831a6862ace.slice/crio-a418d3b996f71c417ddbbc1617f8602cbd82bbb460904c614657ac2bd488d8fe WatchSource:0}: Error finding container a418d3b996f71c417ddbbc1617f8602cbd82bbb460904c614657ac2bd488d8fe: Status 404 returned error can't find the container with id a418d3b996f71c417ddbbc1617f8602cbd82bbb460904c614657ac2bd488d8fe
Jul 29 16:52:14.419361 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.419288    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:14.623142 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.623112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:14.892509 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.892474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tbq6b" event={"ID":"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c","Type":"ContainerStarted","Data":"33b6c260709e283328676edc011d303b751867bdf0117530488314d5b16f1086"}
Jul 29 16:52:14.895314 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.895285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" event={"ID":"edb857fd-9890-4518-818c-af126e650336","Type":"ContainerStarted","Data":"a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39"}
Jul 29 16:52:14.895447 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.895336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" event={"ID":"edb857fd-9890-4518-818c-af126e650336","Type":"ContainerStarted","Data":"0c49542b78289b5058186f4cf217452d33b85e83f4359493e996e5910eb9b9c6"}
Jul 29 16:52:14.896097 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.896060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:14.897682 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.897654    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n5vf" event={"ID":"c061c174-6b97-4f67-b636-6831a6862ace","Type":"ContainerStarted","Data":"a418d3b996f71c417ddbbc1617f8602cbd82bbb460904c614657ac2bd488d8fe"}
Jul 29 16:52:14.917554 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:14.917072    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" podStartSLOduration=31.917052314 podStartE2EDuration="31.917052314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:14.916120897 +0000 UTC m=+52.766154596" watchObservedRunningTime="2026-07-29 16:52:14.917052314 +0000 UTC m=+52.767086014"
Jul 29 16:52:15.203520 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:15.203483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:15.410345 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:15.410305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:15.602318 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:15.602227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:16.911391 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:16.911292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tbq6b" event={"ID":"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c","Type":"ContainerStarted","Data":"345aea4bec568bf029ac19c3d061a8825960e1158e2007aba1172c2fe0f666af"}
Jul 29 16:52:19.923457 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:19.923421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n5vf" event={"ID":"c061c174-6b97-4f67-b636-6831a6862ace","Type":"ContainerStarted","Data":"bc143692bbf5606db05fd178f1fb72ff937455ea9b6f48cb761988ba63a96491"}
Jul 29 16:52:19.925096 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:19.925061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tbq6b" event={"ID":"bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c","Type":"ContainerStarted","Data":"9252165ef3fd0eb24ee226e1306e968310f73a4503392d21bd3b532889e22119"}
Jul 29 16:52:19.925224 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:19.925213    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:19.947988 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:19.947942    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tbq6b" podStartSLOduration=19.529561628 podStartE2EDuration="21.947929514s" totalImagesPullingTime="2.418367886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:19.946645458 +0000 UTC m=+57.796679156" watchObservedRunningTime="2026-07-29 16:52:19.947929514 +0000 UTC m=+57.797963211"
Jul 29 16:52:22.090052 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.090011    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"]
Jul 29 16:52:22.104032 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.104002    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"]
Jul 29 16:52:22.104215 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.104142    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.106851 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.106828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.108469 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.108428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 16:52:22.108597 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.108560    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c5j9b\""
Jul 29 16:52:22.108847 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.108776    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 16:52:22.109573 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.109548    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"]
Jul 29 16:52:22.114828 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.114474    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 16:52:22.114828 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.114702    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 16:52:22.115457 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.115408    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 16:52:22.115631 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.115615    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 16:52:22.115824 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.115802    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"]
Jul 29 16:52:22.115920 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.115888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 16:52:22.116338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.116313    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 16:52:22.127767 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.127745    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b54f7fb5-hvl2p"]
Jul 29 16:52:22.167995 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.167965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-hub\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.168134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae4d5043-74be-43f4-a044-620e4fb3048c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp\" (UID: \"ae4d5043-74be-43f4-a044-620e4fb3048c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.168134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168024    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-ca\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.168134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f658\" (UniqueName: \"kubernetes.io/projected/2fa1efd2-8451-4738-afbb-878481dbf6f3-kube-api-access-8f658\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.168134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.168348 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1efd2-8451-4738-afbb-878481dbf6f3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.168348 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168237    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.168348 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.168303    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvf6\" (UniqueName: \"kubernetes.io/projected/ae4d5043-74be-43f4-a044-620e4fb3048c-kube-api-access-swvf6\") pod \"managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp\" (UID: \"ae4d5043-74be-43f4-a044-620e4fb3048c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.192411 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.192369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:22.264016 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.263974    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56cbcf4895-xtnxt"]
Jul 29 16:52:22.266988 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.266967    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.269352 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.269477 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1efd2-8451-4738-afbb-878481dbf6f3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.269477 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.269477 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvf6\" (UniqueName: \"kubernetes.io/projected/ae4d5043-74be-43f4-a044-620e4fb3048c-kube-api-access-swvf6\") pod \"managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp\" (UID: \"ae4d5043-74be-43f4-a044-620e4fb3048c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.269639 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-hub\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.269639 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae4d5043-74be-43f4-a044-620e4fb3048c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp\" (UID: \"ae4d5043-74be-43f4-a044-620e4fb3048c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.269639 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-ca\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.269639 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.269576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f658\" (UniqueName: \"kubernetes.io/projected/2fa1efd2-8451-4738-afbb-878481dbf6f3-kube-api-access-8f658\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.270757 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.270146    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1efd2-8451-4738-afbb-878481dbf6f3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.270757 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.270512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:22.272058 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.272034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-hub\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.272193 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.272173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.272436 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.272418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.272477 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.272419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae4d5043-74be-43f4-a044-620e4fb3048c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp\" (UID: \"ae4d5043-74be-43f4-a044-620e4fb3048c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.272718 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.272701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2fa1efd2-8451-4738-afbb-878481dbf6f3-ca\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.284337 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.284316    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cbcf4895-xtnxt"]
Jul 29 16:52:22.286767 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.286743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvf6\" (UniqueName: \"kubernetes.io/projected/ae4d5043-74be-43f4-a044-620e4fb3048c-kube-api-access-swvf6\") pod \"managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp\" (UID: \"ae4d5043-74be-43f4-a044-620e4fb3048c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.287820 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.287796    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f658\" (UniqueName: \"kubernetes.io/projected/2fa1efd2-8451-4738-afbb-878481dbf6f3-kube-api-access-8f658\") pod \"cluster-proxy-proxy-agent-66f6b87794-mm2sr\" (UID: \"2fa1efd2-8451-4738-afbb-878481dbf6f3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.370063 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.369985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-registry-tls\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370063 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68555fa0-771f-4771-bc4e-8900251a5071-image-registry-private-configuration\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370273 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-bound-sa-token\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370273 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68555fa0-771f-4771-bc4e-8900251a5071-installation-pull-secrets\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370273 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blf89\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-kube-api-access-blf89\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370273 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68555fa0-771f-4771-bc4e-8900251a5071-trusted-ca\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370273 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68555fa0-771f-4771-bc4e-8900251a5071-ca-trust-extracted\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.370273 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.370275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68555fa0-771f-4771-bc4e-8900251a5071-registry-certificates\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.402385 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.402352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:22.427389 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.427366    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"
Jul 29 16:52:22.433710 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.433684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"
Jul 29 16:52:22.471303 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68555fa0-771f-4771-bc4e-8900251a5071-trusted-ca\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471303 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68555fa0-771f-4771-bc4e-8900251a5071-ca-trust-extracted\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471541 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68555fa0-771f-4771-bc4e-8900251a5071-registry-certificates\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471541 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-registry-tls\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471541 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68555fa0-771f-4771-bc4e-8900251a5071-image-registry-private-configuration\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471541 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-bound-sa-token\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471541 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68555fa0-771f-4771-bc4e-8900251a5071-installation-pull-secrets\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471541 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blf89\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-kube-api-access-blf89\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.471823 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.471787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68555fa0-771f-4771-bc4e-8900251a5071-ca-trust-extracted\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.472393 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.472268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68555fa0-771f-4771-bc4e-8900251a5071-registry-certificates\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.472393 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.472354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68555fa0-771f-4771-bc4e-8900251a5071-trusted-ca\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.474864 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.474817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68555fa0-771f-4771-bc4e-8900251a5071-installation-pull-secrets\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.474864 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.474857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68555fa0-771f-4771-bc4e-8900251a5071-image-registry-private-configuration\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.475031 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.474990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-registry-tls\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.481215 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.481191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-bound-sa-token\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.481926 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.481883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blf89\" (UniqueName: \"kubernetes.io/projected/68555fa0-771f-4771-bc4e-8900251a5071-kube-api-access-blf89\") pod \"image-registry-56cbcf4895-xtnxt\" (UID: \"68555fa0-771f-4771-bc4e-8900251a5071\") " pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.483420 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.483388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:22.599026 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.598997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:22.829982 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:22.829943    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scjn5"
Jul 29 16:52:23.062947 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.062905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:23.103704 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.103634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cbcf4895-xtnxt"]
Jul 29 16:52:23.106831 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:23.106809    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68555fa0_771f_4771_bc4e_8900251a5071.slice/crio-898e2a41988d8d13c8d73b9af696533ee4feb558d30fb7ff5bc743dfae8bff95 WatchSource:0}: Error finding container 898e2a41988d8d13c8d73b9af696533ee4feb558d30fb7ff5bc743dfae8bff95: Status 404 returned error can't find the container with id 898e2a41988d8d13c8d73b9af696533ee4feb558d30fb7ff5bc743dfae8bff95
Jul 29 16:52:23.113287 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.113268    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr"]
Jul 29 16:52:23.119598 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:23.119576    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fa1efd2_8451_4738_afbb_878481dbf6f3.slice/crio-9985ee76296ad9320ba4c4760d322d2b9348655afc5862d51466375cfa9f8b49 WatchSource:0}: Error finding container 9985ee76296ad9320ba4c4760d322d2b9348655afc5862d51466375cfa9f8b49: Status 404 returned error can't find the container with id 9985ee76296ad9320ba4c4760d322d2b9348655afc5862d51466375cfa9f8b49
Jul 29 16:52:23.128738 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.128719    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp"]
Jul 29 16:52:23.136989 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:23.136966    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae4d5043_74be_43f4_a044_620e4fb3048c.slice/crio-e92713b02fc9804553901e769095e3c2de73e3804efa60aa79f91ea13b76f587 WatchSource:0}: Error finding container e92713b02fc9804553901e769095e3c2de73e3804efa60aa79f91ea13b76f587: Status 404 returned error can't find the container with id e92713b02fc9804553901e769095e3c2de73e3804efa60aa79f91ea13b76f587
Jul 29 16:52:23.191282 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.191246    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:23.355510 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.355435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:23.419267 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.419229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:23.540304 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.540272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:23.649987 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.649897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:23.944408 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.944307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n5vf" event={"ID":"c061c174-6b97-4f67-b636-6831a6862ace","Type":"ContainerStarted","Data":"d599f89c06d5ad93f27080fd75f69cd274d3bc42da98e48a14c4a1d85d517813"}
Jul 29 16:52:23.946605 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.946473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt" event={"ID":"68555fa0-771f-4771-bc4e-8900251a5071","Type":"ContainerStarted","Data":"6c69a0b2f7f8a649f9315c883e77d8ca5c7704c947bfcc6fe95f1deaf1524ae2"}
Jul 29 16:52:23.946605 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.946521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt" event={"ID":"68555fa0-771f-4771-bc4e-8900251a5071","Type":"ContainerStarted","Data":"898e2a41988d8d13c8d73b9af696533ee4feb558d30fb7ff5bc743dfae8bff95"}
Jul 29 16:52:23.946884 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.946841    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:23.948228 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.948175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp" event={"ID":"ae4d5043-74be-43f4-a044-620e4fb3048c","Type":"ContainerStarted","Data":"e92713b02fc9804553901e769095e3c2de73e3804efa60aa79f91ea13b76f587"}
Jul 29 16:52:23.949555 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.949513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr" event={"ID":"2fa1efd2-8451-4738-afbb-878481dbf6f3","Type":"ContainerStarted","Data":"9985ee76296ad9320ba4c4760d322d2b9348655afc5862d51466375cfa9f8b49"}
Jul 29 16:52:23.985181 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:23.985051    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt" podStartSLOduration=1.985031238 podStartE2EDuration="1.985031238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:23.981835316 +0000 UTC m=+61.831869018" watchObservedRunningTime="2026-07-29 16:52:23.985031238 +0000 UTC m=+61.835064936"
Jul 29 16:52:24.354603 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:24.354381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:24.437851 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:24.437790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:24.684944 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:24.684881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:24.808951 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:24.808919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:24.874180 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:24.873944    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:26.932192 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:26.932162    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tbq6b"
Jul 29 16:52:28.321562 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.321517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:52:28.324287 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.324263    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 16:52:28.334937 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.334911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c52131-cd16-415a-a195-0d47aca1a9c4-metrics-certs\") pod \"network-metrics-daemon-stljd\" (UID: \"f1c52131-cd16-415a-a195-0d47aca1a9c4\") " pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:52:28.371014 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.370968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:28.420817 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.420786    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fgvc\""
Jul 29 16:52:28.422066 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.422037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:52:28.424545 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.424528    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 16:52:28.428716 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.428698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stljd"
Jul 29 16:52:28.434684 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.434661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f51ad380-8ca6-47ae-be18-66d23531f26f-original-pull-secret\") pod \"global-pull-secret-syncer-gkqgp\" (UID: \"f51ad380-8ca6-47ae-be18-66d23531f26f\") " pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:52:28.472960 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.472924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:28.522637 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.522605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:52:28.525416 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.525394    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 16:52:28.532043 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.532008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:28.536572 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.536552    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 16:52:28.546830 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.546804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlnx\" (UniqueName: \"kubernetes.io/projected/a1c31620-e5ed-42b4-a0b1-e6b412624dc6-kube-api-access-6jlnx\") pod \"network-check-target-ntfhp\" (UID: \"a1c31620-e5ed-42b4-a0b1-e6b412624dc6\") " pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:52:28.712556 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.712526    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qtk2\""
Jul 29 16:52:28.720274 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.720240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:52:28.723095 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:28.723063    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkqgp"
Jul 29 16:52:29.459719 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.459584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ntfhp"]
Jul 29 16:52:29.463185 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:29.463146    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c31620_e5ed_42b4_a0b1_e6b412624dc6.slice/crio-862bd4d42ba971e1d82546910fdc16f40c4035459033c11776f48820da52a814 WatchSource:0}: Error finding container 862bd4d42ba971e1d82546910fdc16f40c4035459033c11776f48820da52a814: Status 404 returned error can't find the container with id 862bd4d42ba971e1d82546910fdc16f40c4035459033c11776f48820da52a814
Jul 29 16:52:29.476698 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.476647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stljd"]
Jul 29 16:52:29.491232 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:29.491202    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c52131_cd16_415a_a195_0d47aca1a9c4.slice/crio-30c8bf90881b710e783014bf83176766364234b66a6b8e6207a0a3fc950d6dab WatchSource:0}: Error finding container 30c8bf90881b710e783014bf83176766364234b66a6b8e6207a0a3fc950d6dab: Status 404 returned error can't find the container with id 30c8bf90881b710e783014bf83176766364234b66a6b8e6207a0a3fc950d6dab
Jul 29 16:52:29.677696 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.677664    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gkqgp"]
Jul 29 16:52:29.682041 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:29.682010    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51ad380_8ca6_47ae_be18_66d23531f26f.slice/crio-5865242a9c7a2d1ecb3b2c66554a716b724b7e0778d8ac247050fc7bb1a73917 WatchSource:0}: Error finding container 5865242a9c7a2d1ecb3b2c66554a716b724b7e0778d8ac247050fc7bb1a73917: Status 404 returned error can't find the container with id 5865242a9c7a2d1ecb3b2c66554a716b724b7e0778d8ac247050fc7bb1a73917
Jul 29 16:52:29.975156 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.975121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gkqgp" event={"ID":"f51ad380-8ca6-47ae-be18-66d23531f26f","Type":"ContainerStarted","Data":"5865242a9c7a2d1ecb3b2c66554a716b724b7e0778d8ac247050fc7bb1a73917"}
Jul 29 16:52:29.976575 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.976542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ntfhp" event={"ID":"a1c31620-e5ed-42b4-a0b1-e6b412624dc6","Type":"ContainerStarted","Data":"862bd4d42ba971e1d82546910fdc16f40c4035459033c11776f48820da52a814"}
Jul 29 16:52:29.978839 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.978812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n5vf" event={"ID":"c061c174-6b97-4f67-b636-6831a6862ace","Type":"ContainerStarted","Data":"73b5b0d6e6a3f30b240fbd65f3dbe6ddb576f4b65d0349972fd07cf6ae069cad"}
Jul 29 16:52:29.980402 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.980379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp" event={"ID":"ae4d5043-74be-43f4-a044-620e4fb3048c","Type":"ContainerStarted","Data":"3ebd9248cd7c868e59faeee00e0c058c75fd6af8abfd6f359d3b847d300ae3be"}
Jul 29 16:52:29.982647 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.982619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr" event={"ID":"2fa1efd2-8451-4738-afbb-878481dbf6f3","Type":"ContainerStarted","Data":"5224edbec946aee24a6ce3cbef557888274c4a3a55de2a679a35f383606fa0be"}
Jul 29 16:52:29.982752 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.982656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr" event={"ID":"2fa1efd2-8451-4738-afbb-878481dbf6f3","Type":"ContainerStarted","Data":"a961e349e71c11b5e6642d25053db52d871fde22a5babf2ab3841a6a295918ee"}
Jul 29 16:52:29.982752 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.982673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr" event={"ID":"2fa1efd2-8451-4738-afbb-878481dbf6f3","Type":"ContainerStarted","Data":"cdbc6b3c87f734de7643465fc055f0b204bb3b448f459a8ab39c52cf9701ee80"}
Jul 29 16:52:29.983847 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:29.983822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stljd" event={"ID":"f1c52131-cd16-415a-a195-0d47aca1a9c4","Type":"ContainerStarted","Data":"30c8bf90881b710e783014bf83176766364234b66a6b8e6207a0a3fc950d6dab"}
Jul 29 16:52:30.000908 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:30.000866    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6n5vf" podStartSLOduration=17.210560124 podStartE2EDuration="32.000851444s" totalImagesPullingTime="14.79029132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:29.998946801 +0000 UTC m=+67.848980498" watchObservedRunningTime="2026-07-29 16:52:30.000851444 +0000 UTC m=+67.850885143"
Jul 29 16:52:30.038612 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:30.038562    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ccfc4d4b8-nf7lp" podStartSLOduration=1.91474988 podStartE2EDuration="8.038545078s" totalImagesPullingTime="6.123795198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:30.014984065 +0000 UTC m=+67.865017766" watchObservedRunningTime="2026-07-29 16:52:30.038545078 +0000 UTC m=+67.888578778"
Jul 29 16:52:30.038730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:30.038657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66f6b87794-mm2sr" podStartSLOduration=1.886502323 podStartE2EDuration="8.038650636s" totalImagesPullingTime="6.152148313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:30.0375617 +0000 UTC m=+67.887595394" watchObservedRunningTime="2026-07-29 16:52:30.038650636 +0000 UTC m=+67.888684339"
Jul 29 16:52:30.643880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:30.643847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:30.712950 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:30.712893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:30.993642 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:30.993611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stljd" event={"ID":"f1c52131-cd16-415a-a195-0d47aca1a9c4","Type":"ContainerStarted","Data":"22fda3adf38421b235d591556483c0eadf261b80e76c6073795c1804d6f7b11f"}
Jul 29 16:52:31.618629 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:31.618592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:31.719985 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:31.719948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:31.999941 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:31.999896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stljd" event={"ID":"f1c52131-cd16-415a-a195-0d47aca1a9c4","Type":"ContainerStarted","Data":"cfd38a7cebff588bed281eb853f67e8d48548203979f84a774f677059f9077e9"}
Jul 29 16:52:32.017792 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:32.017737    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-stljd" podStartSLOduration=68.646041724 podStartE2EDuration="1m10.01771711s" totalImagesPullingTime="1.371675386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:32.01631217 +0000 UTC m=+69.866345869" watchObservedRunningTime="2026-07-29 16:52:32.01771711 +0000 UTC m=+69.867750811"
Jul 29 16:52:32.136212 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:32.136177    2559 patch_prober.go:28] interesting pod/image-registry-57b54f7fb5-hvl2p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 16:52:32.136369 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:32.136234    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" podUID="edb857fd-9890-4518-818c-af126e650336" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 16:52:32.198544 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:32.198489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:32.355659 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:32.355534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:32.527482 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:32.527435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:33.002592 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.002558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:33.006449 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.006407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ntfhp" event={"ID":"a1c31620-e5ed-42b4-a0b1-e6b412624dc6","Type":"ContainerStarted","Data":"95df8fcd9d3bfe9b2f00a0f7c6892326a7847c42e27f58d394b038ee0cbd048d"}
Jul 29 16:52:33.006601 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.006486    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:52:33.024526 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.024477    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ntfhp" podStartSLOduration=66.952182234 podStartE2EDuration="1m10.024459762s" totalImagesPullingTime="3.072277528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:33.024452984 +0000 UTC m=+70.874486683" watchObservedRunningTime="2026-07-29 16:52:33.024459762 +0000 UTC m=+70.874493462"
Jul 29 16:52:33.273883 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.273787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:33.397625 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.397591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:33.509627 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:33.509586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:34.801207 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:34.801117    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:34.897335 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:34.897301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:35.013961 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:35.013931    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gkqgp" event={"ID":"f51ad380-8ca6-47ae-be18-66d23531f26f","Type":"ContainerStarted","Data":"da48c23377dd94051e9b213b1a73012bd0d87bc8208ce020505a2cb23374a5a6"}
Jul 29 16:52:35.292025 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:35.291995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:35.361808 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:35.361779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:35.728549 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:35.728507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:35.939259 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:35.939230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:36.041017 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:36.040951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:36.140403 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:36.140371    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:38.458670 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:38.458642    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:38.559750 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:38.559709    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:38.799192 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:38.799121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:40.987550 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:40.987517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:41.091854 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:41.091815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:42.132096 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:42.132050    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:42.155897 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:42.155842    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gkqgp" podStartSLOduration=74.409657725 podStartE2EDuration="1m19.155825049s" totalImagesPullingTime="4.746167324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:35.032398104 +0000 UTC m=+72.882431802" watchObservedRunningTime="2026-07-29 16:52:42.155825049 +0000 UTC m=+80.005858749"
Jul 29 16:52:42.669811 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:42.669778    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:42.762613 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:42.762583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:42.862624 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:42.862588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:42.923134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:42.923061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:43.266159 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:43.266125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:44.368229 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:44.368195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:44.471025 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:44.470914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:44.959604 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:44.959577    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56cbcf4895-xtnxt"
Jul 29 16:52:45.111044 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.111012    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:45.214367 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.214291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:45.598643 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.598562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:45.865406 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.865313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:45.961904 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.961825    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fdstd"]
Jul 29 16:52:45.966747 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.966723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:45.973716 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.973681    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 16:52:45.974673 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.974656    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 16:52:45.975245 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.974849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 16:52:45.975349 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.975252    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 16:52:45.975405 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.975102    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 16:52:45.975465 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.975023    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltjzg\""
Jul 29 16:52:45.975535 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.975173    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 16:52:45.980469 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:45.980444    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:46.064904 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.064871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.064904 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.064901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxngp\" (UniqueName: \"kubernetes.io/projected/7a592f16-4bb0-4558-913c-7a60d49b945d-kube-api-access-sxngp\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.064919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-sys\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.064941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a592f16-4bb0-4558-913c-7a60d49b945d-metrics-client-ca\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.064975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-textfile\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.065000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-root\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.065020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.065053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-wtmp\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.065183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.065075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-tls\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.118194 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.118103    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:46.166125 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166125 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxngp\" (UniqueName: \"kubernetes.io/projected/7a592f16-4bb0-4558-913c-7a60d49b945d-kube-api-access-sxngp\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-sys\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a592f16-4bb0-4558-913c-7a60d49b945d-metrics-client-ca\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-textfile\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-root\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166338 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-wtmp\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166620 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-tls\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166620 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-sys\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166620 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-root\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166620 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:46.166450    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 16:52:46.166620 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:46.166520    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-tls podName:7a592f16-4bb0-4558-913c-7a60d49b945d nodeName:}" failed. No retries permitted until 2026-07-29 16:52:46.666498519 +0000 UTC m=+84.516532196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-tls") pod "node-exporter-fdstd" (UID: "7a592f16-4bb0-4558-913c-7a60d49b945d") : secret "node-exporter-tls" not found
Jul 29 16:52:46.166855 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-wtmp\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166855 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-textfile\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166855 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.166855 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.166839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a592f16-4bb0-4558-913c-7a60d49b945d-metrics-client-ca\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.168662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.168638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.176822 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.176802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxngp\" (UniqueName: \"kubernetes.io/projected/7a592f16-4bb0-4558-913c-7a60d49b945d-kube-api-access-sxngp\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.366999 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.366961    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:46.467122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.467071    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:46.670732 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.670699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-tls\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.672982 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.672962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a592f16-4bb0-4558-913c-7a60d49b945d-node-exporter-tls\") pod \"node-exporter-fdstd\" (UID: \"7a592f16-4bb0-4558-913c-7a60d49b945d\") " pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.772872 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.772798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:46.878039 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.878008    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fdstd"
Jul 29 16:52:46.889243 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:52:46.889217    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a592f16_4bb0_4558_913c_7a60d49b945d.slice/crio-b40b130cfa8089030b69b37adc7c2da5b39108f22e330485c2b70187e639f7ea WatchSource:0}: Error finding container b40b130cfa8089030b69b37adc7c2da5b39108f22e330485c2b70187e639f7ea: Status 404 returned error can't find the container with id b40b130cfa8089030b69b37adc7c2da5b39108f22e330485c2b70187e639f7ea
Jul 29 16:52:46.891930 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.891900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:46.981119 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:46.981070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:47.059952 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.059869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdstd" event={"ID":"7a592f16-4bb0-4558-913c-7a60d49b945d","Type":"ContainerStarted","Data":"b40b130cfa8089030b69b37adc7c2da5b39108f22e330485c2b70187e639f7ea"}
Jul 29 16:52:47.147766 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.147734    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" podUID="edb857fd-9890-4518-818c-af126e650336" containerName="registry" containerID="cri-o://a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39" gracePeriod=30
Jul 29 16:52:47.284940 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.284905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:47.404844 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.404819    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:47.578515 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-installation-pull-secrets\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578515 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578468    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-bound-sa-token\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-registry-certificates\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578572    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edb857fd-9890-4518-818c-af126e650336-ca-trust-extracted\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578596    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-image-registry-private-configuration\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578657    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-trusted-ca\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.578728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.578690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcbv4\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-kube-api-access-mcbv4\") pod \"edb857fd-9890-4518-818c-af126e650336\" (UID: \"edb857fd-9890-4518-818c-af126e650336\") "
Jul 29 16:52:47.579099 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.579048    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:52:47.579552 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.579513    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 16:52:47.581943 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.581894    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:52:47.582047 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.581951    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:52:47.582170 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.582141    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:52:47.582303 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.582274    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-kube-api-access-mcbv4" (OuterVolumeSpecName: "kube-api-access-mcbv4") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "kube-api-access-mcbv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:52:47.582303 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.582285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 16:52:47.589970 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.589944    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb857fd-9890-4518-818c-af126e650336-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "edb857fd-9890-4518-818c-af126e650336" (UID: "edb857fd-9890-4518-818c-af126e650336"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 16:52:47.679997 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.679969    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-registry-certificates\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.679997 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.679994    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edb857fd-9890-4518-818c-af126e650336-ca-trust-extracted\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.679997 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.680004    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-registry-tls\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.680405 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.680013    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-image-registry-private-configuration\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.680405 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.680023    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edb857fd-9890-4518-818c-af126e650336-trusted-ca\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.680405 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.680031    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbv4\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-kube-api-access-mcbv4\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.680405 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.680039    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edb857fd-9890-4518-818c-af126e650336-installation-pull-secrets\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:47.680405 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:47.680051    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edb857fd-9890-4518-818c-af126e650336-bound-sa-token\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:52:48.064762 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.064727    2559 generic.go:356] "Generic (PLEG): container finished" podID="edb857fd-9890-4518-818c-af126e650336" containerID="a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39" exitCode=0
Jul 29 16:52:48.065027 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.064774    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p"
Jul 29 16:52:48.065027 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.064811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" event={"ID":"edb857fd-9890-4518-818c-af126e650336","Type":"ContainerDied","Data":"a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39"}
Jul 29 16:52:48.065027 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.064870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b54f7fb5-hvl2p" event={"ID":"edb857fd-9890-4518-818c-af126e650336","Type":"ContainerDied","Data":"0c49542b78289b5058186f4cf217452d33b85e83f4359493e996e5910eb9b9c6"}
Jul 29 16:52:48.065027 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.064900    2559 scope.go:117] "RemoveContainer" containerID="a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39"
Jul 29 16:52:48.066589 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.066568    2559 generic.go:356] "Generic (PLEG): container finished" podID="7a592f16-4bb0-4558-913c-7a60d49b945d" containerID="19e1defdd829dcba1628a5ad5392149a646e9ef69700d63e60a8165f9ee7b2a6" exitCode=0
Jul 29 16:52:48.066681 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.066659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdstd" event={"ID":"7a592f16-4bb0-4558-913c-7a60d49b945d","Type":"ContainerDied","Data":"19e1defdd829dcba1628a5ad5392149a646e9ef69700d63e60a8165f9ee7b2a6"}
Jul 29 16:52:48.076252 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.076237    2559 scope.go:117] "RemoveContainer" containerID="a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39"
Jul 29 16:52:48.076493 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:52:48.076475    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39\": container with ID starting with a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39 not found: ID does not exist" containerID="a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39"
Jul 29 16:52:48.076538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.076501    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39"} err="failed to get container status \"a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39\": rpc error: code = NotFound desc = could not find container \"a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39\": container with ID starting with a30171144db39404a9d25d588a0cb57c24216c5d4b0d8489c2b5c9cac3bede39 not found: ID does not exist"
Jul 29 16:52:48.107519 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.107499    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b54f7fb5-hvl2p"]
Jul 29 16:52:48.111886 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.111863    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57b54f7fb5-hvl2p"]
Jul 29 16:52:48.703856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.703819    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb857fd-9890-4518-818c-af126e650336" path="/var/lib/kubelet/pods/edb857fd-9890-4518-818c-af126e650336/volumes"
Jul 29 16:52:48.934896 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:48.934862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:49.033288 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:49.033194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:49.072850 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:49.072810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdstd" event={"ID":"7a592f16-4bb0-4558-913c-7a60d49b945d","Type":"ContainerStarted","Data":"64bbe6bbd29be774f1f4695992605dde0afdd4f933e79602b01661b018862dc6"}
Jul 29 16:52:49.073001 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:49.072869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdstd" event={"ID":"7a592f16-4bb0-4558-913c-7a60d49b945d","Type":"ContainerStarted","Data":"9320a5ed89656ad718723e7b7505cd4cecef5a8503c91d58a7d334f888e100e3"}
Jul 29 16:52:49.097869 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:49.097823    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fdstd" podStartSLOduration=3.210172062 podStartE2EDuration="4.097807772s" totalImagesPullingTime="824.70467ms" totalInitContainerRuntime="62.93104ms" isStatefulPod=true podCreationTimestamp="2026-07-29 16:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:52:49.095861353 +0000 UTC m=+86.945895051" watchObservedRunningTime="2026-07-29 16:52:49.097807772 +0000 UTC m=+86.947841469"
Jul 29 16:52:49.232688 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:49.232639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:49.336755 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:49.336670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:50.699854 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:50.699822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:50.775600 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:50.775562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:50.847298 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:50.847266    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:50.949036 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:50.948996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:51.167131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:51.167038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:51.239603 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:51.239564    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:52.305499 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:52.305463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:52.458981 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:52.458943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:52.560393 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:52.560313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:55.805970 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:55.805939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:55.903528 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:55.903498    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:58.386488 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:58.386455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:52:58.474204 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:52:58.474172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:04.012207 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:04.012174    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ntfhp"
Jul 29 16:53:06.817679 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:06.817651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:07.426514 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:07.426476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:07.487317 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:07.487278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:13.221630 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:13.221593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:13.373972 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:13.373941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:13.437913 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:13.437874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:53:49.096540 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:53:49.096503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:02.176070 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:02.176033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:02.266789 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:02.266754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:08.681533 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:08.681485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:14.172847 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:14.172809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:14.239131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:14.239100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:20.106234 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:20.106201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:20.209256 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:20.209221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:20.271347 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:20.271318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:28.372973 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:28.372939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:28.470909 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:28.470867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:54:56.825965 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:54:56.825923    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:55:49.648012 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:55:49.647905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:55:49.738604 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:55:49.738548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:55:49.799473 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:55:49.799439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:17.930116 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:17.930057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:21.569870 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:21.569832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:22.507344 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:22.507319    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 16:56:23.712311 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:23.712269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.001949 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.001869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.359752 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.359649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.461073 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.461032    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.664378 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.664291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.764750 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.764707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.868805 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.868772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:24.889017 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.888987    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-4n2jr"]
Jul 29 16:56:24.889294 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.889279    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb857fd-9890-4518-818c-af126e650336" containerName="registry"
Jul 29 16:56:24.889368 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.889297    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb857fd-9890-4518-818c-af126e650336" containerName="registry"
Jul 29 16:56:24.889421 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.889366    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb857fd-9890-4518-818c-af126e650336" containerName="registry"
Jul 29 16:56:24.891973 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.891953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:24.894861 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.894839    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 29 16:56:24.894861 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.894858    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 29 16:56:24.895820 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.895795    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 29 16:56:24.895820 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.895814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-bn7k2\""
Jul 29 16:56:24.895951 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.895833    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 29 16:56:24.909856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.909830    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-4n2jr"]
Jul 29 16:56:24.932955 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:24.932924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.023047 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.023011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19eb619b-e882-4ec7-b8a3-ef4f8389996f-certificates\") pod \"keda-admission-56f9798bb4-4n2jr\" (UID: \"19eb619b-e882-4ec7-b8a3-ef4f8389996f\") " pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.023223 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.023075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2b4z\" (UniqueName: \"kubernetes.io/projected/19eb619b-e882-4ec7-b8a3-ef4f8389996f-kube-api-access-f2b4z\") pod \"keda-admission-56f9798bb4-4n2jr\" (UID: \"19eb619b-e882-4ec7-b8a3-ef4f8389996f\") " pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.076183 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.076144    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.123691 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.123662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19eb619b-e882-4ec7-b8a3-ef4f8389996f-certificates\") pod \"keda-admission-56f9798bb4-4n2jr\" (UID: \"19eb619b-e882-4ec7-b8a3-ef4f8389996f\") " pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.123867 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.123707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2b4z\" (UniqueName: \"kubernetes.io/projected/19eb619b-e882-4ec7-b8a3-ef4f8389996f-kube-api-access-f2b4z\") pod \"keda-admission-56f9798bb4-4n2jr\" (UID: \"19eb619b-e882-4ec7-b8a3-ef4f8389996f\") " pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.126417 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.126395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19eb619b-e882-4ec7-b8a3-ef4f8389996f-certificates\") pod \"keda-admission-56f9798bb4-4n2jr\" (UID: \"19eb619b-e882-4ec7-b8a3-ef4f8389996f\") " pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.133468 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.133442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2b4z\" (UniqueName: \"kubernetes.io/projected/19eb619b-e882-4ec7-b8a3-ef4f8389996f-kube-api-access-f2b4z\") pod \"keda-admission-56f9798bb4-4n2jr\" (UID: \"19eb619b-e882-4ec7-b8a3-ef4f8389996f\") " pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.180305 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.180276    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.203131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.203058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:25.285764 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.285727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.339753 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.339612    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-4n2jr"]
Jul 29 16:56:25.342338 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:56:25.342301    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eb619b_e882_4ec7_b8a3_ef4f8389996f.slice/crio-68bbfecefd1a91580e8e96dcd92c0769aeee3e6bc8702693b3780b755982e7f9 WatchSource:0}: Error finding container 68bbfecefd1a91580e8e96dcd92c0769aeee3e6bc8702693b3780b755982e7f9: Status 404 returned error can't find the container with id 68bbfecefd1a91580e8e96dcd92c0769aeee3e6bc8702693b3780b755982e7f9
Jul 29 16:56:25.343678 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.343656    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 16:56:25.475882 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.475789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.634131 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.634095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.742982 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.742898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:25.762170 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:25.762142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr" event={"ID":"19eb619b-e882-4ec7-b8a3-ef4f8389996f","Type":"ContainerStarted","Data":"68bbfecefd1a91580e8e96dcd92c0769aeee3e6bc8702693b3780b755982e7f9"}
Jul 29 16:56:29.778011 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:29.777977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr" event={"ID":"19eb619b-e882-4ec7-b8a3-ef4f8389996f","Type":"ContainerStarted","Data":"ede01e1c2c1ec58ced5a84b551ab17e9537710763e99722cdcb709be9ee27bac"}
Jul 29 16:56:29.778450 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:29.778216    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:56:29.795891 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:29.795848    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr" podStartSLOduration=1.882580253 podStartE2EDuration="5.795835113s" totalImagesPullingTime="3.91325486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:56:29.794854369 +0000 UTC m=+307.644888067" watchObservedRunningTime="2026-07-29 16:56:29.795835113 +0000 UTC m=+307.645868811"
Jul 29 16:56:49.702245 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:49.702214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:49.765013 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:49.764973    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:56:50.783552 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:56:50.783521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-4n2jr"
Jul 29 16:57:16.087835 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:16.087733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:16.192967 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:16.192932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:27.476007 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:27.475977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:28.767646 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:28.767609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:31.306905 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:31.306868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:31.406538 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:31.406507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:33.626894 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:33.626861    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:33.721575 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:33.721540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:35.806854 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:35.806822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:36.224419 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:36.224387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:37.363232 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:37.363199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:37.465239 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:37.465197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:37.639623 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:37.639539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:37.731171 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:37.731143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:37.835187 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:37.835151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:44.422322 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:44.422286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:45.548425 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:45.548384    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:45.649141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:45.649066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:49.529098 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:49.529048    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:57:49.623773 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:57:49.623727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:01.035516 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:01.035433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:05.891941 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:05.891905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:05.956290 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:05.956250    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:07.051593 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:07.051557    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:08.854876 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:08.854842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:09.882976 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:09.882946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:09.983209 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:09.983177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:12.537898 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.537869    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"]
Jul 29 16:58:12.541206 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.541187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.543972 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.543949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-zqjl2\""
Jul 29 16:58:12.544231 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.544216    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 16:58:12.544337 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.544282    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 16:58:12.544337 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.544285    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 16:58:12.554238 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.554220    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"]
Jul 29 16:58:12.579529 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.579502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:12.671825 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6v86\" (UniqueName: \"kubernetes.io/projected/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-kube-api-access-h6v86\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-workload-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-data\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-workload-certs\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671971    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672000 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.671995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-token\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.672271 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.672023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-credential-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.673029 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.673004    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:12.772759 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-credential-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.772896 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.772896 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6v86\" (UniqueName: \"kubernetes.io/projected/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-kube-api-access-h6v86\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.772896 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-workload-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.772896 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-data\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-workload-certs\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773122 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.772987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-token\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773318 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.773222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-credential-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773318 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.773306    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-workload-certs\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773422 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.773342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-workload-socket\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773526 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.773503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-data\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.773714 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.773696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.775162 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.775143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.775427 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.775412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.781450 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.781427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-istio-token\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.781670 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.781642    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6v86\" (UniqueName: \"kubernetes.io/projected/153d0a1f-89b0-4f64-a1ee-47f3f5c06953-kube-api-access-h6v86\") pod \"openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs\" (UID: \"153d0a1f-89b0-4f64-a1ee-47f3f5c06953\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.850693 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.850627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:12.976857 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:12.976823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"]
Jul 29 16:58:12.979544 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:58:12.979517    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153d0a1f_89b0_4f64_a1ee_47f3f5c06953.slice/crio-292f113e25fc608e666edcca121b9409ca15c7632be421e1dc3577cb7dec6517 WatchSource:0}: Error finding container 292f113e25fc608e666edcca121b9409ca15c7632be421e1dc3577cb7dec6517: Status 404 returned error can't find the container with id 292f113e25fc608e666edcca121b9409ca15c7632be421e1dc3577cb7dec6517
Jul 29 16:58:13.118551 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:13.118455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" event={"ID":"153d0a1f-89b0-4f64-a1ee-47f3f5c06953","Type":"ContainerStarted","Data":"292f113e25fc608e666edcca121b9409ca15c7632be421e1dc3577cb7dec6517"}
Jul 29 16:58:15.843559 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:15.843524    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 16:58:15.843797 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:15.843595    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 16:58:15.843797 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:15.843628    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 16:58:16.130807 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.130725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" event={"ID":"153d0a1f-89b0-4f64-a1ee-47f3f5c06953","Type":"ContainerStarted","Data":"f10b620ee607f347822817be88db008ab5906963092c76605fc626f6054131d4"}
Jul 29 16:58:16.153277 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.153231    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podStartSLOduration=1.291284263 podStartE2EDuration="4.153216916s" totalImagesPullingTime="2.861932653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:58:16.152228578 +0000 UTC m=+414.002262274" watchObservedRunningTime="2026-07-29 16:58:16.153216916 +0000 UTC m=+414.003250613"
Jul 29 16:58:16.279397 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.279365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:16.379207 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.379174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:16.466956 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.466931    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"]
Jul 29 16:58:16.469987 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.469971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.472952 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.472936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 16:58:16.473178 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.473156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 16:58:16.473226 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.473184    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 16:58:16.473589 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.473573    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 16:58:16.473634 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.473573    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 16:58:16.473892 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.473876    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-drjj8\""
Jul 29 16:58:16.476828 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.476808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:16.552611 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.552590    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"]
Jul 29 16:58:16.602728 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.602706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95c90d10-8080-4856-8938-6c822078e54f-cert\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.602818 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.602748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95c90d10-8080-4856-8938-6c822078e54f-metrics-cert\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.602856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.602808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95c90d10-8080-4856-8938-6c822078e54f-manager-config\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.602856 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.602840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwpv\" (UniqueName: \"kubernetes.io/projected/95c90d10-8080-4856-8938-6c822078e54f-kube-api-access-qhwpv\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.703508 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.703486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95c90d10-8080-4856-8938-6c822078e54f-manager-config\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.703607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.703516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwpv\" (UniqueName: \"kubernetes.io/projected/95c90d10-8080-4856-8938-6c822078e54f-kube-api-access-qhwpv\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.703607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.703544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95c90d10-8080-4856-8938-6c822078e54f-cert\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.703607 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.703580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95c90d10-8080-4856-8938-6c822078e54f-metrics-cert\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.704243 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.704223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95c90d10-8080-4856-8938-6c822078e54f-manager-config\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.706003 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.705978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95c90d10-8080-4856-8938-6c822078e54f-cert\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.706066 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.706036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95c90d10-8080-4856-8938-6c822078e54f-metrics-cert\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.715051 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.715025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwpv\" (UniqueName: \"kubernetes.io/projected/95c90d10-8080-4856-8938-6c822078e54f-kube-api-access-qhwpv\") pod \"lws-controller-manager-6bb667976b-pzw89\" (UID: \"95c90d10-8080-4856-8938-6c822078e54f\") " pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.779459 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.779388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:16.851772 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.851741    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:16.856911 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.856879    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:16.919752 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:16.919725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"]
Jul 29 16:58:16.922500 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:58:16.922474    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c90d10_8080_4856_8938_6c822078e54f.slice/crio-b530fb6e4bd89074d7c4b7cbde69e3299c361f41eeb2ec8e6fdc0a54f3e9e66c WatchSource:0}: Error finding container b530fb6e4bd89074d7c4b7cbde69e3299c361f41eeb2ec8e6fdc0a54f3e9e66c: Status 404 returned error can't find the container with id b530fb6e4bd89074d7c4b7cbde69e3299c361f41eeb2ec8e6fdc0a54f3e9e66c
Jul 29 16:58:17.135185 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:17.135090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89" event={"ID":"95c90d10-8080-4856-8938-6c822078e54f","Type":"ContainerStarted","Data":"b530fb6e4bd89074d7c4b7cbde69e3299c361f41eeb2ec8e6fdc0a54f3e9e66c"}
Jul 29 16:58:17.135406 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:17.135389    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:17.136308 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:17.136291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 16:58:18.105516 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:18.105423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:18.204610 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:18.204578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:20.148428 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:20.148395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89" event={"ID":"95c90d10-8080-4856-8938-6c822078e54f","Type":"ContainerStarted","Data":"a3635738c403f92bb2077bb1f6d2d0980b9260a23839552fb7af0bedc57bf14e"}
Jul 29 16:58:20.148814 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:20.148525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:20.173704 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:20.173657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89" podStartSLOduration=1.651477573 podStartE2EDuration="4.17364311s" totalImagesPullingTime="2.522165537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:58:20.171691858 +0000 UTC m=+418.021725584" watchObservedRunningTime="2026-07-29 16:58:20.17364311 +0000 UTC m=+418.023676808"
Jul 29 16:58:20.294165 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:20.294125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:20.392215 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:20.392185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:20.492975 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:20.492947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:25.047783 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:25.047749    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:31.154490 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:31.154458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bb667976b-pzw89"
Jul 29 16:58:31.990326 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:31.990286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:32.765897 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:32.765859    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:35.820011 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:35.819971    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:35.922971 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:35.922938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:36.014975 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:36.014936    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:37.234820 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:37.234789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:38.359146 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:38.359113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:38.620680 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:38.620591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:40.375348 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:40.375308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:40.756829 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:40.756797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:41.160048 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:41.159977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:41.483515 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:41.483481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:41.586767 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:41.586735    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:45.575648 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:45.575551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:46.277310 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:46.277279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:46.385680 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:46.385643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:48.571929 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:48.571895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:48.638923 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:48.638894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:51.298523 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.298492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:51.309721 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.309692    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"]
Jul 29 16:58:51.317148 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.317128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.319762 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.319739    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wmsnf\""
Jul 29 16:58:51.319880 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.319743    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 16:58:51.320353 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.320328    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 16:58:51.320649 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.320622    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 16:58:51.320739 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.320658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"]
Jul 29 16:58:51.320811 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.320772    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 16:58:51.352649 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.352625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0349333e-24ad-43a2-a6c2-8bf576b49e1c-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.352744 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.352659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0349333e-24ad-43a2-a6c2-8bf576b49e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.352744 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.352705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5jl\" (UniqueName: \"kubernetes.io/projected/0349333e-24ad-43a2-a6c2-8bf576b49e1c-kube-api-access-bd5jl\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.396322 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.396298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:51.453023 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.453000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0349333e-24ad-43a2-a6c2-8bf576b49e1c-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.453136 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.453035    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0349333e-24ad-43a2-a6c2-8bf576b49e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.453136 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.453097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5jl\" (UniqueName: \"kubernetes.io/projected/0349333e-24ad-43a2-a6c2-8bf576b49e1c-kube-api-access-bd5jl\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.453727 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.453709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0349333e-24ad-43a2-a6c2-8bf576b49e1c-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.455385 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.455364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0349333e-24ad-43a2-a6c2-8bf576b49e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.465633 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.465609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5jl\" (UniqueName: \"kubernetes.io/projected/0349333e-24ad-43a2-a6c2-8bf576b49e1c-kube-api-access-bd5jl\") pod \"kuadrant-console-plugin-598f86468b-89xrd\" (UID: \"0349333e-24ad-43a2-a6c2-8bf576b49e1c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.627324 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.627267    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"
Jul 29 16:58:51.764545 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:51.764429    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd"]
Jul 29 16:58:51.767159 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:58:51.767131    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0349333e_24ad_43a2_a6c2_8bf576b49e1c.slice/crio-bb2c0f56829d1a44caab776b7413e0b345f17815cbdb4e578b651d1d081e670e WatchSource:0}: Error finding container bb2c0f56829d1a44caab776b7413e0b345f17815cbdb4e578b651d1d081e670e: Status 404 returned error can't find the container with id bb2c0f56829d1a44caab776b7413e0b345f17815cbdb4e578b651d1d081e670e
Jul 29 16:58:52.260134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:52.260104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd" event={"ID":"0349333e-24ad-43a2-a6c2-8bf576b49e1c","Type":"ContainerStarted","Data":"bb2c0f56829d1a44caab776b7413e0b345f17815cbdb4e578b651d1d081e670e"}
Jul 29 16:58:52.836387 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:52.836331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:52.936655 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:52.936602    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:57.279963 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:57.279930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd" event={"ID":"0349333e-24ad-43a2-a6c2-8bf576b49e1c","Type":"ContainerStarted","Data":"66344fc39d6a085a6283be988ec67b46661eb764b49438d3936653f89c8ba72d"}
Jul 29 16:58:57.296632 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:57.296587    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-89xrd" podStartSLOduration=1.696522446 podStartE2EDuration="6.296572129s" totalImagesPullingTime="4.600049683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:58:57.296046641 +0000 UTC m=+455.146080340" watchObservedRunningTime="2026-07-29 16:58:57.296572129 +0000 UTC m=+455.146605827"
Jul 29 16:58:57.585040 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:57.584959    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:57.685409 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:57.685382    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:58:57.750455 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:58:57.750427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:04.646320 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:04.646277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:04.746120 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:04.746007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:08.099494 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:08.099456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:08.199953 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:08.199921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:08.532008 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:08.531968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:08.600473 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:08.600442    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:12.271635 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:12.271600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:12.383881 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:12.383846    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:16.864915 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:16.864853    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:16.961876 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:16.961843    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:17.062407 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:17.062362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:35.949875 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:35.949831    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:36.057314 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.057278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:36.160788 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.160745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:36.803863 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.803831    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-t6c5n"]
Jul 29 16:59:36.823460 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.823432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-t6c5n"]
Jul 29 16:59:36.823593 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.823514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:36.826196 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.826176    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9lzdc\""
Jul 29 16:59:36.830820 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.830791    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:36.891134 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.891104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4nd9\" (UniqueName: \"kubernetes.io/projected/1e5f290f-4084-425b-a315-6ce7972230bc-kube-api-access-b4nd9\") pod \"authorino-b54df94b4-t6c5n\" (UID: \"1e5f290f-4084-425b-a315-6ce7972230bc\") " pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:36.930781 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.930741    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:36.931351 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.931334    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqsxt"]
Jul 29 16:59:36.934377 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.934362    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 16:59:36.944798 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.944777    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqsxt"]
Jul 29 16:59:36.992513 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:36.992483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4nd9\" (UniqueName: \"kubernetes.io/projected/1e5f290f-4084-425b-a315-6ce7972230bc-kube-api-access-b4nd9\") pod \"authorino-b54df94b4-t6c5n\" (UID: \"1e5f290f-4084-425b-a315-6ce7972230bc\") " pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:37.005920 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.005895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4nd9\" (UniqueName: \"kubernetes.io/projected/1e5f290f-4084-425b-a315-6ce7972230bc-kube-api-access-b4nd9\") pod \"authorino-b54df94b4-t6c5n\" (UID: \"1e5f290f-4084-425b-a315-6ce7972230bc\") " pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:37.093150 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.093049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqzw\" (UniqueName: \"kubernetes.io/projected/31e28278-e966-4574-8083-91ce8154b6a6-kube-api-access-pkqzw\") pod \"authorino-c548b5d7d-mqsxt\" (UID: \"31e28278-e966-4574-8083-91ce8154b6a6\") " pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 16:59:37.133093 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.133056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:37.194730 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.194427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqzw\" (UniqueName: \"kubernetes.io/projected/31e28278-e966-4574-8083-91ce8154b6a6-kube-api-access-pkqzw\") pod \"authorino-c548b5d7d-mqsxt\" (UID: \"31e28278-e966-4574-8083-91ce8154b6a6\") " pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 16:59:37.203614 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.203571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqzw\" (UniqueName: \"kubernetes.io/projected/31e28278-e966-4574-8083-91ce8154b6a6-kube-api-access-pkqzw\") pod \"authorino-c548b5d7d-mqsxt\" (UID: \"31e28278-e966-4574-8083-91ce8154b6a6\") " pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 16:59:37.242250 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.242219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 16:59:37.256494 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.256363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-t6c5n"]
Jul 29 16:59:37.259036 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:59:37.259004    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5f290f_4084_425b_a315_6ce7972230bc.slice/crio-9e61cf154e781a900e01e9741d17938facbf07644881a661b1bf4495fda9993f WatchSource:0}: Error finding container 9e61cf154e781a900e01e9741d17938facbf07644881a661b1bf4495fda9993f: Status 404 returned error can't find the container with id 9e61cf154e781a900e01e9741d17938facbf07644881a661b1bf4495fda9993f
Jul 29 16:59:37.365107 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.365015    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqsxt"]
Jul 29 16:59:37.367979 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:59:37.367954    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31e28278_e966_4574_8083_91ce8154b6a6.slice/crio-d41dd6de3a1644eec6f55388cffb269f4dc53b6681f578df3d7248af65ad0e88 WatchSource:0}: Error finding container d41dd6de3a1644eec6f55388cffb269f4dc53b6681f578df3d7248af65ad0e88: Status 404 returned error can't find the container with id d41dd6de3a1644eec6f55388cffb269f4dc53b6681f578df3d7248af65ad0e88
Jul 29 16:59:37.421572 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.421534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqsxt" event={"ID":"31e28278-e966-4574-8083-91ce8154b6a6","Type":"ContainerStarted","Data":"d41dd6de3a1644eec6f55388cffb269f4dc53b6681f578df3d7248af65ad0e88"}
Jul 29 16:59:37.422580 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:37.422562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-t6c5n" event={"ID":"1e5f290f-4084-425b-a315-6ce7972230bc","Type":"ContainerStarted","Data":"9e61cf154e781a900e01e9741d17938facbf07644881a661b1bf4495fda9993f"}
Jul 29 16:59:40.435779 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:40.435743    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqsxt" event={"ID":"31e28278-e966-4574-8083-91ce8154b6a6","Type":"ContainerStarted","Data":"c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701"}
Jul 29 16:59:40.437266 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:40.437240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-t6c5n" event={"ID":"1e5f290f-4084-425b-a315-6ce7972230bc","Type":"ContainerStarted","Data":"18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1"}
Jul 29 16:59:40.452310 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:40.452229    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-mqsxt" podStartSLOduration=1.604736973 podStartE2EDuration="4.452214326s" totalImagesPullingTime="2.847477353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:59:40.450949775 +0000 UTC m=+498.300983473" watchObservedRunningTime="2026-07-29 16:59:40.452214326 +0000 UTC m=+498.302248034"
Jul 29 16:59:40.488332 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:40.488285    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-t6c5n" podStartSLOduration=1.541289251 podStartE2EDuration="4.488270318s" totalImagesPullingTime="2.946981067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:59:40.464898281 +0000 UTC m=+498.314931979" watchObservedRunningTime="2026-07-29 16:59:40.488270318 +0000 UTC m=+498.338304018"
Jul 29 16:59:40.488477 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:40.488463    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-t6c5n"]
Jul 29 16:59:42.447788 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:42.447751    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-t6c5n" podUID="1e5f290f-4084-425b-a315-6ce7972230bc" containerName="authorino" containerID="cri-o://18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1" gracePeriod=30
Jul 29 16:59:42.695975 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:42.695950    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:42.843179 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:42.843075    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4nd9\" (UniqueName: \"kubernetes.io/projected/1e5f290f-4084-425b-a315-6ce7972230bc-kube-api-access-b4nd9\") pod \"1e5f290f-4084-425b-a315-6ce7972230bc\" (UID: \"1e5f290f-4084-425b-a315-6ce7972230bc\") "
Jul 29 16:59:42.845253 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:42.845225    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5f290f-4084-425b-a315-6ce7972230bc-kube-api-access-b4nd9" (OuterVolumeSpecName: "kube-api-access-b4nd9") pod "1e5f290f-4084-425b-a315-6ce7972230bc" (UID: "1e5f290f-4084-425b-a315-6ce7972230bc"). InnerVolumeSpecName "kube-api-access-b4nd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 16:59:42.943662 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:42.943622    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4nd9\" (UniqueName: \"kubernetes.io/projected/1e5f290f-4084-425b-a315-6ce7972230bc-kube-api-access-b4nd9\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 16:59:43.451984 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.451946    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e5f290f-4084-425b-a315-6ce7972230bc" containerID="18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1" exitCode=0
Jul 29 16:59:43.452544 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.452004    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-t6c5n"
Jul 29 16:59:43.452544 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.452032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-t6c5n" event={"ID":"1e5f290f-4084-425b-a315-6ce7972230bc","Type":"ContainerDied","Data":"18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1"}
Jul 29 16:59:43.452544 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.452102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-t6c5n" event={"ID":"1e5f290f-4084-425b-a315-6ce7972230bc","Type":"ContainerDied","Data":"9e61cf154e781a900e01e9741d17938facbf07644881a661b1bf4495fda9993f"}
Jul 29 16:59:43.452544 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.452127    2559 scope.go:117] "RemoveContainer" containerID="18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1"
Jul 29 16:59:43.459965 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.459946    2559 scope.go:117] "RemoveContainer" containerID="18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1"
Jul 29 16:59:43.460222 ip-10-0-139-76 kubenswrapper[2559]: E0729 16:59:43.460204    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1\": container with ID starting with 18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1 not found: ID does not exist" containerID="18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1"
Jul 29 16:59:43.460282 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.460229    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1"} err="failed to get container status \"18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1\": rpc error: code = NotFound desc = could not find container \"18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1\": container with ID starting with 18c04dfb2e3cb50bd8df569c6e0b0b785c782fda97f78284fb32bb9acd7f18d1 not found: ID does not exist"
Jul 29 16:59:43.473879 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.473858    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-t6c5n"]
Jul 29 16:59:43.481267 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:43.481241    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-t6c5n"]
Jul 29 16:59:44.703718 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:44.703687    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e5f290f-4084-425b-a315-6ce7972230bc" path="/var/lib/kubelet/pods/1e5f290f-4084-425b-a315-6ce7972230bc/volumes"
Jul 29 16:59:55.665387 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:55.665356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:55.944282 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:55.944250    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 16:59:59.154360 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.154321    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-867c7768fd-xj254"]
Jul 29 16:59:59.154739 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.154576    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5f290f-4084-425b-a315-6ce7972230bc" containerName="authorino"
Jul 29 16:59:59.154739 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.154587    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5f290f-4084-425b-a315-6ce7972230bc" containerName="authorino"
Jul 29 16:59:59.154739 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.154633    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5f290f-4084-425b-a315-6ce7972230bc" containerName="authorino"
Jul 29 16:59:59.157412 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.157396    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.161653 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.161626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h4w\" (UniqueName: \"kubernetes.io/projected/f09bac5c-ea26-48fa-ae61-6f453c4a8506-kube-api-access-d5h4w\") pod \"authorino-867c7768fd-xj254\" (UID: \"f09bac5c-ea26-48fa-ae61-6f453c4a8506\") " pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.161779 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.161668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f09bac5c-ea26-48fa-ae61-6f453c4a8506-tls-cert\") pod \"authorino-867c7768fd-xj254\" (UID: \"f09bac5c-ea26-48fa-ae61-6f453c4a8506\") " pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.161864 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.161849    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 16:59:59.185141 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.185115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-867c7768fd-xj254"]
Jul 29 16:59:59.263068 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.263038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h4w\" (UniqueName: \"kubernetes.io/projected/f09bac5c-ea26-48fa-ae61-6f453c4a8506-kube-api-access-d5h4w\") pod \"authorino-867c7768fd-xj254\" (UID: \"f09bac5c-ea26-48fa-ae61-6f453c4a8506\") " pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.263234 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.263072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f09bac5c-ea26-48fa-ae61-6f453c4a8506-tls-cert\") pod \"authorino-867c7768fd-xj254\" (UID: \"f09bac5c-ea26-48fa-ae61-6f453c4a8506\") " pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.265387 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.265359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f09bac5c-ea26-48fa-ae61-6f453c4a8506-tls-cert\") pod \"authorino-867c7768fd-xj254\" (UID: \"f09bac5c-ea26-48fa-ae61-6f453c4a8506\") " pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.272814 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.272793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h4w\" (UniqueName: \"kubernetes.io/projected/f09bac5c-ea26-48fa-ae61-6f453c4a8506-kube-api-access-d5h4w\") pod \"authorino-867c7768fd-xj254\" (UID: \"f09bac5c-ea26-48fa-ae61-6f453c4a8506\") " pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.466329 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.466290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-867c7768fd-xj254"
Jul 29 16:59:59.596388 ip-10-0-139-76 kubenswrapper[2559]: I0729 16:59:59.596226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-867c7768fd-xj254"]
Jul 29 16:59:59.598805 ip-10-0-139-76 kubenswrapper[2559]: W0729 16:59:59.598780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf09bac5c_ea26_48fa_ae61_6f453c4a8506.slice/crio-0395c8b272a51e16837f6b2318ee40a81ed806146ca8f7ac6d3163504769eacf WatchSource:0}: Error finding container 0395c8b272a51e16837f6b2318ee40a81ed806146ca8f7ac6d3163504769eacf: Status 404 returned error can't find the container with id 0395c8b272a51e16837f6b2318ee40a81ed806146ca8f7ac6d3163504769eacf
Jul 29 17:00:00.514356 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:00.514331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-867c7768fd-xj254" event={"ID":"f09bac5c-ea26-48fa-ae61-6f453c4a8506","Type":"ContainerStarted","Data":"0395c8b272a51e16837f6b2318ee40a81ed806146ca8f7ac6d3163504769eacf"}
Jul 29 17:00:01.519547 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.519512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-867c7768fd-xj254" event={"ID":"f09bac5c-ea26-48fa-ae61-6f453c4a8506","Type":"ContainerStarted","Data":"b4d39930292488316b076541d68ac1fac1b060ee9e4fd3b7dc961b4939a52d91"}
Jul 29 17:00:01.543638 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.543600    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-867c7768fd-xj254" podStartSLOduration=1.675267093 podStartE2EDuration="2.543585217s" totalImagesPullingTime="868.318124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:00:01.543384954 +0000 UTC m=+519.393418652" watchObservedRunningTime="2026-07-29 17:00:01.543585217 +0000 UTC m=+519.393618915"
Jul 29 17:00:01.572966 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.572939    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqsxt"]
Jul 29 17:00:01.573179 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.573156    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-mqsxt" podUID="31e28278-e966-4574-8083-91ce8154b6a6" containerName="authorino" containerID="cri-o://c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701" gracePeriod=30
Jul 29 17:00:01.809419 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.809393    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 17:00:01.880855 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.880824    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkqzw\" (UniqueName: \"kubernetes.io/projected/31e28278-e966-4574-8083-91ce8154b6a6-kube-api-access-pkqzw\") pod \"31e28278-e966-4574-8083-91ce8154b6a6\" (UID: \"31e28278-e966-4574-8083-91ce8154b6a6\") "
Jul 29 17:00:01.882706 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.882682    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e28278-e966-4574-8083-91ce8154b6a6-kube-api-access-pkqzw" (OuterVolumeSpecName: "kube-api-access-pkqzw") pod "31e28278-e966-4574-8083-91ce8154b6a6" (UID: "31e28278-e966-4574-8083-91ce8154b6a6"). InnerVolumeSpecName "kube-api-access-pkqzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:00:01.981323 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:01.981301    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqzw\" (UniqueName: \"kubernetes.io/projected/31e28278-e966-4574-8083-91ce8154b6a6-kube-api-access-pkqzw\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:00:02.524322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.524281    2559 generic.go:356] "Generic (PLEG): container finished" podID="31e28278-e966-4574-8083-91ce8154b6a6" containerID="c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701" exitCode=0
Jul 29 17:00:02.524783 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.524335    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqsxt"
Jul 29 17:00:02.524783 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.524365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqsxt" event={"ID":"31e28278-e966-4574-8083-91ce8154b6a6","Type":"ContainerDied","Data":"c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701"}
Jul 29 17:00:02.524783 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.524404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqsxt" event={"ID":"31e28278-e966-4574-8083-91ce8154b6a6","Type":"ContainerDied","Data":"d41dd6de3a1644eec6f55388cffb269f4dc53b6681f578df3d7248af65ad0e88"}
Jul 29 17:00:02.524783 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.524426    2559 scope.go:117] "RemoveContainer" containerID="c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701"
Jul 29 17:00:02.534300 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.534284    2559 scope.go:117] "RemoveContainer" containerID="c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701"
Jul 29 17:00:02.534711 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:00:02.534679    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701\": container with ID starting with c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701 not found: ID does not exist" containerID="c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701"
Jul 29 17:00:02.534802 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.534718    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701"} err="failed to get container status \"c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701\": rpc error: code = NotFound desc = could not find container \"c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701\": container with ID starting with c81f27a413aa94f4ac58474b2ca72a484b47b3c35b9ce8fcefd75a2b4bab6701 not found: ID does not exist"
Jul 29 17:00:02.546346 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.546323    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqsxt"]
Jul 29 17:00:02.549931 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.549913    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqsxt"]
Jul 29 17:00:02.702917 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:02.702894    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e28278-e966-4574-8083-91ce8154b6a6" path="/var/lib/kubelet/pods/31e28278-e966-4574-8083-91ce8154b6a6/volumes"
Jul 29 17:00:04.618217 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:04.618181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:04.718877 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:04.718845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:19.568761 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:19.568675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:19.649317 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:19.649282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:19.741211 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:19.741177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:20.020465 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:20.020431    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:20.236359 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:20.236329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:20.378275 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:20.378192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:20.479241 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:20.479206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:36.856145 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:36.856106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:00:36.920009 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:00:36.919977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:17.840727 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:17.840696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:17.946131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:17.946098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:32.851356 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:32.851320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:34.436843 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:34.436804    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:39.010919 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:39.010882    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:49.074022 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:49.073938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:49.136216 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:49.136180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:59.715146 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:59.715114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:01:59.805743 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:01:59.805708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:01.013287 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:01.013193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:01.113945 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:01.113915    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:07.835269 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:07.835230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:07.951925 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:07.951885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:09.749394 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.749355    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"]
Jul 29 17:02:09.749813 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.749760    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e28278-e966-4574-8083-91ce8154b6a6" containerName="authorino"
Jul 29 17:02:09.749813 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.749777    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e28278-e966-4574-8083-91ce8154b6a6" containerName="authorino"
Jul 29 17:02:09.749890 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.749849    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e28278-e966-4574-8083-91ce8154b6a6" containerName="authorino"
Jul 29 17:02:09.752786 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.752770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.755629 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.755610    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 17:02:09.755730 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.755610    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-f5l4h\""
Jul 29 17:02:09.755886 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.755872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 17:02:09.756777 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.756762    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 29 17:02:09.767160 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.767135    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"]
Jul 29 17:02:09.783059 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.783034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:09.794507 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-credential-socket\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-podinfo\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-data\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794704 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-workload-socket\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794704 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-workload-certs\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794704 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-token\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794795 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-envoy\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.794795 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.794723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxvj\" (UniqueName: \"kubernetes.io/projected/581f680c-a2a0-4443-b3ca-4834dfaa9c17-kube-api-access-2sxvj\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.883280 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.883250    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:09.895182 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-envoy\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895260 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxvj\" (UniqueName: \"kubernetes.io/projected/581f680c-a2a0-4443-b3ca-4834dfaa9c17-kube-api-access-2sxvj\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895260 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-credential-socket\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895260 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-podinfo\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895405 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-data\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895405 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895405 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895364    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-workload-socket\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895535 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-workload-certs\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895535 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-token\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895674 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-credential-socket\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895763 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895740    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-data\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.895856 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.895830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-workload-socket\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.896036 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.896017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-workload-certs\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.896155 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.896135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.897457 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.897438    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-envoy\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.897649 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.897630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-podinfo\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.905895 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.905865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581f680c-a2a0-4443-b3ca-4834dfaa9c17-istio-token\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:09.906104 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:09.906065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxvj\" (UniqueName: \"kubernetes.io/projected/581f680c-a2a0-4443-b3ca-4834dfaa9c17-kube-api-access-2sxvj\") pod \"router-gateway-1-openshift-default-579df5d64f-t9b75\" (UID: \"581f680c-a2a0-4443-b3ca-4834dfaa9c17\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:10.065767 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.065713    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:10.196868 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.196735    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"]
Jul 29 17:02:10.198996 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:02:10.198967    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581f680c_a2a0_4443_b3ca_4834dfaa9c17.slice/crio-2d432ada467e314e2d9df0ac8d6b9c000f120b895983566aa771d0c9c3d6bb79 WatchSource:0}: Error finding container 2d432ada467e314e2d9df0ac8d6b9c000f120b895983566aa771d0c9c3d6bb79: Status 404 returned error can't find the container with id 2d432ada467e314e2d9df0ac8d6b9c000f120b895983566aa771d0c9c3d6bb79
Jul 29 17:02:10.200756 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.200724    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:10.200835 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.200786    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:10.200835 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.200816    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:10.964802 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.964769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" event={"ID":"581f680c-a2a0-4443-b3ca-4834dfaa9c17","Type":"ContainerStarted","Data":"ebfcf01235c4bc0e1dddc30a23511012cc1535619493feb3b02950a6a1b755f1"}
Jul 29 17:02:10.965300 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.964820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" event={"ID":"581f680c-a2a0-4443-b3ca-4834dfaa9c17","Type":"ContainerStarted","Data":"2d432ada467e314e2d9df0ac8d6b9c000f120b895983566aa771d0c9c3d6bb79"}
Jul 29 17:02:10.990675 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:10.990632    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podStartSLOduration=1.990618261 podStartE2EDuration="1.990618261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:02:10.99021563 +0000 UTC m=+648.840249328" watchObservedRunningTime="2026-07-29 17:02:10.990618261 +0000 UTC m=+648.840651958"
Jul 29 17:02:11.065979 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:11.065937    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:11.070655 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:11.070631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:11.968914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:11.968877    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:11.969711 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:11.969693    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:13.468262 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:13.468228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:22.055811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:22.055743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:22.585901 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:22.585864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:32.136638 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:32.136599    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Jul 29 17:02:32.136962 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:32.136668    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podUID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Jul 29 17:02:33.045171 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:33.045132    2559 generic.go:356] "Generic (PLEG): container finished" podID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerID="f10b620ee607f347822817be88db008ab5906963092c76605fc626f6054131d4" exitCode=0
Jul 29 17:02:33.045342 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:33.045204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" event={"ID":"153d0a1f-89b0-4f64-a1ee-47f3f5c06953","Type":"ContainerDied","Data":"f10b620ee607f347822817be88db008ab5906963092c76605fc626f6054131d4"}
Jul 29 17:02:33.045613 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:33.045598    2559 scope.go:117] "RemoveContainer" containerID="f10b620ee607f347822817be88db008ab5906963092c76605fc626f6054131d4"
Jul 29 17:02:33.046631 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:33.046597    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:33.046701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:33.046684    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:33.046739 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:33.046724    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:34.054119 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:34.053314    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" event={"ID":"153d0a1f-89b0-4f64-a1ee-47f3f5c06953","Type":"ContainerStarted","Data":"01833d912cce1bae7bc9fce423c33f994a4cfc3da036b113258b1892840485ce"}
Jul 29 17:02:34.850936 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:34.850900    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 17:02:34.851521 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:34.851493    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Jul 29 17:02:34.851774 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:34.851724    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podUID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Jul 29 17:02:35.851052 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:35.851014    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Jul 29 17:02:35.852035 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:35.851626    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podUID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Jul 29 17:02:36.575784 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.573148    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"]
Jul 29 17:02:36.576844 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.576816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.580671 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.580639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 29 17:02:36.580783 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.580675    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:02:36.589399 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.589377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"]
Jul 29 17:02:36.607072 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.607048    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:36.714942 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.714915    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.715116 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.714954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.715116 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.714974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.715116 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.715012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.715116 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.715037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gs6\" (UniqueName: \"kubernetes.io/projected/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kube-api-access-42gs6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.715270 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.715125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.715270 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.715160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816195 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816364 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816364 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816364 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816364 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gs6\" (UniqueName: \"kubernetes.io/projected/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kube-api-access-42gs6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816579 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816642 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816714 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816772 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816758    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.816810 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.817008 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.816985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.818838 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.818817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.818929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.818844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.824173 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.824151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gs6\" (UniqueName: \"kubernetes.io/projected/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kube-api-access-42gs6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:36.851369 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.851306    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Jul 29 17:02:36.852174 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.851801    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podUID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Jul 29 17:02:36.887044 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:36.887022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:37.056106 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:37.055773    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"]
Jul 29 17:02:37.073818 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:02:37.073770    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a91ead_b4dc_4ece_8cde_8925bb5d0929.slice/crio-f3f08218ec05fd75c857c36fd49d4b8332674a05feb487dcaabac8109f73382e WatchSource:0}: Error finding container f3f08218ec05fd75c857c36fd49d4b8332674a05feb487dcaabac8109f73382e: Status 404 returned error can't find the container with id f3f08218ec05fd75c857c36fd49d4b8332674a05feb487dcaabac8109f73382e
Jul 29 17:02:37.075830 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:37.075813    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 17:02:37.864107 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:37.861516    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Jul 29 17:02:37.864107 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:37.861576    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podUID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Jul 29 17:02:38.097027 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:38.096980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9" event={"ID":"e5a91ead-b4dc-4ece-8cde-8925bb5d0929","Type":"ContainerStarted","Data":"f3f08218ec05fd75c857c36fd49d4b8332674a05feb487dcaabac8109f73382e"}
Jul 29 17:02:38.869054 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:38.868566    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body=
Jul 29 17:02:38.869054 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:38.868633    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs" podUID="153d0a1f-89b0-4f64-a1ee-47f3f5c06953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused"
Jul 29 17:02:39.873017 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:39.872981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 17:02:39.873479 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:39.873428    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 17:02:39.874244 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:39.874223    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs"
Jul 29 17:02:42.116039 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:42.116003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9" event={"ID":"e5a91ead-b4dc-4ece-8cde-8925bb5d0929","Type":"ContainerStarted","Data":"0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58"}
Jul 29 17:02:44.701151 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:44.699652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:45.743618 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:45.743575    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"]
Jul 29 17:02:45.744051 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:45.743854    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9" podUID="e5a91ead-b4dc-4ece-8cde-8925bb5d0929" containerName="storage-initializer" containerID="cri-o://0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58" gracePeriod=30
Jul 29 17:02:45.802519 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:45.802481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:02:46.006196 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.006135    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:46.122749 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122709    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-model-cache\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.122931 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122758    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-dshm\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.122931 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gs6\" (UniqueName: \"kubernetes.io/projected/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kube-api-access-42gs6\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.122931 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122856    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tmp-dir\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.122931 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122898    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-home\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.123180 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122949    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kserve-provision-location\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.123180 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122975    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:02:46.123180 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.123107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:02:46.123180 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.122983    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tls-certs\") pod \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\" (UID: \"e5a91ead-b4dc-4ece-8cde-8925bb5d0929\") "
Jul 29 17:02:46.123371 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.123272    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-home" (OuterVolumeSpecName: "home") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:02:46.123538 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.123511    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-model-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.123538 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.123534    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tmp-dir\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.123661 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.123549    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-home\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.125050 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.125027    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-dshm" (OuterVolumeSpecName: "dshm") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:02:46.125312 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.125288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kube-api-access-42gs6" (OuterVolumeSpecName: "kube-api-access-42gs6") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "kube-api-access-42gs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:02:46.125586 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.125560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:02:46.136028 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.135998    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5a91ead-b4dc-4ece-8cde-8925bb5d0929" containerID="0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58" exitCode=0
Jul 29 17:02:46.136164 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.136051    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"
Jul 29 17:02:46.136164 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.136058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9" event={"ID":"e5a91ead-b4dc-4ece-8cde-8925bb5d0929","Type":"ContainerDied","Data":"0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58"}
Jul 29 17:02:46.136164 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.136128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9" event={"ID":"e5a91ead-b4dc-4ece-8cde-8925bb5d0929","Type":"ContainerDied","Data":"f3f08218ec05fd75c857c36fd49d4b8332674a05feb487dcaabac8109f73382e"}
Jul 29 17:02:46.136164 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.136158    2559 scope.go:117] "RemoveContainer" containerID="0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58"
Jul 29 17:02:46.206666 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.206599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5a91ead-b4dc-4ece-8cde-8925bb5d0929" (UID: "e5a91ead-b4dc-4ece-8cde-8925bb5d0929"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:02:46.218194 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.218165    2559 scope.go:117] "RemoveContainer" containerID="0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58"
Jul 29 17:02:46.218576 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:02:46.218552    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58\": container with ID starting with 0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58 not found: ID does not exist" containerID="0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58"
Jul 29 17:02:46.218639 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.218590    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58"} err="failed to get container status \"0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58\": rpc error: code = NotFound desc = could not find container \"0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58\": container with ID starting with 0015d82769e3ba47cc9292ca5b9594daebff85269d45d2a4549b9bd0babf7b58 not found: ID does not exist"
Jul 29 17:02:46.223920 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.223893    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.223920 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.223917    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.224055 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.223930    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-dshm\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.224055 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.223939    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gs6\" (UniqueName: \"kubernetes.io/projected/e5a91ead-b4dc-4ece-8cde-8925bb5d0929-kube-api-access-42gs6\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:02:46.471469 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.471443    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"]
Jul 29 17:02:46.474236 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.474210    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8f74bff4c-24tm9"]
Jul 29 17:02:46.704239 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:46.704166    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a91ead-b4dc-4ece-8cde-8925bb5d0929" path="/var/lib/kubelet/pods/e5a91ead-b4dc-4ece-8cde-8925bb5d0929/volumes"
Jul 29 17:02:47.141479 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:47.141388    2559 generic.go:356] "Generic (PLEG): container finished" podID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerID="ebfcf01235c4bc0e1dddc30a23511012cc1535619493feb3b02950a6a1b755f1" exitCode=0
Jul 29 17:02:47.141479 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:47.141462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" event={"ID":"581f680c-a2a0-4443-b3ca-4834dfaa9c17","Type":"ContainerDied","Data":"ebfcf01235c4bc0e1dddc30a23511012cc1535619493feb3b02950a6a1b755f1"}
Jul 29 17:02:47.141986 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:47.141843    2559 scope.go:117] "RemoveContainer" containerID="ebfcf01235c4bc0e1dddc30a23511012cc1535619493feb3b02950a6a1b755f1"
Jul 29 17:02:47.142770 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:47.142741    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:47.142841 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:47.142824    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:47.142891 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:47.142875    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:02:48.147541 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:48.147510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" event={"ID":"581f680c-a2a0-4443-b3ca-4834dfaa9c17","Type":"ContainerStarted","Data":"0712c856e3c1d4cdd08e78c1ee9666caa4309294842ff2337b55d0443b566d00"}
Jul 29 17:02:49.066546 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:49.066508    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:49.070061 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:49.070041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:49.151259 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:49.151234    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:02:49.152402 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:02:49.152381    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:03:07.477417 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.477383    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"]
Jul 29 17:03:07.477788 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.477660    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a91ead-b4dc-4ece-8cde-8925bb5d0929" containerName="storage-initializer"
Jul 29 17:03:07.477788 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.477672    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a91ead-b4dc-4ece-8cde-8925bb5d0929" containerName="storage-initializer"
Jul 29 17:03:07.477788 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.477728    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a91ead-b4dc-4ece-8cde-8925bb5d0929" containerName="storage-initializer"
Jul 29 17:03:07.479779 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.479762    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.482487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.482434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:03:07.482574 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.482485    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 29 17:03:07.484752 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b481619-8792-4c88-b866-73985a965137-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.484874 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484761    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.484874 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4s6\" (UniqueName: \"kubernetes.io/projected/7b481619-8792-4c88-b866-73985a965137-kube-api-access-cq4s6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.484990 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.484990 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.484990 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484947    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.485147 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.484989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.491017 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.490935    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"]
Jul 29 17:03:07.498481 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.498446    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:07.585671 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.585671 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4s6\" (UniqueName: \"kubernetes.io/projected/7b481619-8792-4c88-b866-73985a965137-kube-api-access-cq4s6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.585908 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.585908 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.585908 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.585908 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.586116 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.585956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b481619-8792-4c88-b866-73985a965137-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.586116 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.586074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.586228 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.586137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.586228 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.586183    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.586367 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.586346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.592684 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.592651    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.593136 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.593111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b481619-8792-4c88-b866-73985a965137-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.595223 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.595195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4s6\" (UniqueName: \"kubernetes.io/projected/7b481619-8792-4c88-b866-73985a965137-kube-api-access-cq4s6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.790296 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.790218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:07.939363 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:07.939327    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"]
Jul 29 17:03:07.942401 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:03:07.942374    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b481619_8792_4c88_b866_73985a965137.slice/crio-714d4bcd558d980c42981f08064b869a08e80c6934bcd058ae6dc57ac6fa6611 WatchSource:0}: Error finding container 714d4bcd558d980c42981f08064b869a08e80c6934bcd058ae6dc57ac6fa6611: Status 404 returned error can't find the container with id 714d4bcd558d980c42981f08064b869a08e80c6934bcd058ae6dc57ac6fa6611
Jul 29 17:03:08.220125 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:08.220057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" event={"ID":"7b481619-8792-4c88-b866-73985a965137","Type":"ContainerStarted","Data":"2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8"}
Jul 29 17:03:08.220311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:08.220148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" event={"ID":"7b481619-8792-4c88-b866-73985a965137","Type":"ContainerStarted","Data":"714d4bcd558d980c42981f08064b869a08e80c6934bcd058ae6dc57ac6fa6611"}
Jul 29 17:03:08.378194 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:08.378158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:13.240393 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:13.240355    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b481619-8792-4c88-b866-73985a965137" containerID="2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8" exitCode=0
Jul 29 17:03:13.240872 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:13.240467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" event={"ID":"7b481619-8792-4c88-b866-73985a965137","Type":"ContainerDied","Data":"2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8"}
Jul 29 17:03:15.251730 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:15.251689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" event={"ID":"7b481619-8792-4c88-b866-73985a965137","Type":"ContainerStarted","Data":"4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0"}
Jul 29 17:03:15.273099 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:15.273044    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" podStartSLOduration=2.503478186 podStartE2EDuration="8.273028332s" totalImagesPullingTime="1.153330557s" totalInitContainerRuntime="4.616219589s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:03:15.272746822 +0000 UTC m=+713.122780524" watchObservedRunningTime="2026-07-29 17:03:15.273028332 +0000 UTC m=+713.123062075"
Jul 29 17:03:17.261978 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.261896    2559 generic.go:356] "Generic (PLEG): container finished" podID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerID="0712c856e3c1d4cdd08e78c1ee9666caa4309294842ff2337b55d0443b566d00" exitCode=0
Jul 29 17:03:17.261978 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.261965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" event={"ID":"581f680c-a2a0-4443-b3ca-4834dfaa9c17","Type":"ContainerDied","Data":"0712c856e3c1d4cdd08e78c1ee9666caa4309294842ff2337b55d0443b566d00"}
Jul 29 17:03:17.262513 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.262027    2559 scope.go:117] "RemoveContainer" containerID="ebfcf01235c4bc0e1dddc30a23511012cc1535619493feb3b02950a6a1b755f1"
Jul 29 17:03:17.262513 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.262381    2559 scope.go:117] "RemoveContainer" containerID="0712c856e3c1d4cdd08e78c1ee9666caa4309294842ff2337b55d0443b566d00"
Jul 29 17:03:17.262605 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:03:17.262549    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-579df5d64f-t9b75_kserve-ci-e2e-test(581f680c-a2a0-4443-b3ca-4834dfaa9c17)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17"
Jul 29 17:03:17.790656 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.790614    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:17.790840 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.790682    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:17.803320 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:17.803294    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:18.278336 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:18.278302    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:19.151866 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:19.151831    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:03:19.152265 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:19.152249    2559 scope.go:117] "RemoveContainer" containerID="0712c856e3c1d4cdd08e78c1ee9666caa4309294842ff2337b55d0443b566d00"
Jul 29 17:03:19.152456 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:03:19.152436    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-579df5d64f-t9b75_kserve-ci-e2e-test(581f680c-a2a0-4443-b3ca-4834dfaa9c17)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17"
Jul 29 17:03:26.048191 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.048158    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"]
Jul 29 17:03:26.082142 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.082066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"]
Jul 29 17:03:26.082284 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.082169    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.082493 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.082463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:26.084962 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.084940    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
Jul 29 17:03:26.128048 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f291298b-f689-4106-9459-f25a05cfde76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.128172 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.128172 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.128172 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.128273 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.128273 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfvh\" (UniqueName: \"kubernetes.io/projected/f291298b-f689-4106-9459-f25a05cfde76-kube-api-access-wmfvh\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.128336 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.128275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229250 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229404 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229404 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229404 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229554 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfvh\" (UniqueName: \"kubernetes.io/projected/f291298b-f689-4106-9459-f25a05cfde76-kube-api-access-wmfvh\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229554 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229646 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f291298b-f689-4106-9459-f25a05cfde76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229741 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229741 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229883 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229796    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.229883 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.229837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.231788 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.231762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.231960 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.231943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f291298b-f689-4106-9459-f25a05cfde76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.240295 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.240275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfvh\" (UniqueName: \"kubernetes.io/projected/f291298b-f689-4106-9459-f25a05cfde76-kube-api-access-wmfvh\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.392603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.392502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:26.540345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:26.540321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"]
Jul 29 17:03:26.542173 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:03:26.542134    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf291298b_f689_4106_9459_f25a05cfde76.slice/crio-ec92d0cfcf36c249759508c6dd2320a0e80cae1b0e7dacc867fa92614ee4919e WatchSource:0}: Error finding container ec92d0cfcf36c249759508c6dd2320a0e80cae1b0e7dacc867fa92614ee4919e: Status 404 returned error can't find the container with id ec92d0cfcf36c249759508c6dd2320a0e80cae1b0e7dacc867fa92614ee4919e
Jul 29 17:03:27.299589 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:27.299543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" event={"ID":"f291298b-f689-4106-9459-f25a05cfde76","Type":"ContainerStarted","Data":"3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce"}
Jul 29 17:03:27.300013 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:27.299605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" event={"ID":"f291298b-f689-4106-9459-f25a05cfde76","Type":"ContainerStarted","Data":"ec92d0cfcf36c249759508c6dd2320a0e80cae1b0e7dacc867fa92614ee4919e"}
Jul 29 17:03:27.936136 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:27.936100    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"]
Jul 29 17:03:27.936457 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:27.936404    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" podUID="7b481619-8792-4c88-b866-73985a965137" containerName="main" containerID="cri-o://4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0" gracePeriod=30
Jul 29 17:03:27.949549 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:27.949517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:28.229721 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.229695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:28.308490 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.308455    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b481619-8792-4c88-b866-73985a965137" containerID="4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0" exitCode=0
Jul 29 17:03:28.308929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.308519    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"
Jul 29 17:03:28.308929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.308547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" event={"ID":"7b481619-8792-4c88-b866-73985a965137","Type":"ContainerDied","Data":"4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0"}
Jul 29 17:03:28.308929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.308606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg" event={"ID":"7b481619-8792-4c88-b866-73985a965137","Type":"ContainerDied","Data":"714d4bcd558d980c42981f08064b869a08e80c6934bcd058ae6dc57ac6fa6611"}
Jul 29 17:03:28.308929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.308638    2559 scope.go:117] "RemoveContainer" containerID="4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0"
Jul 29 17:03:28.318324 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.318306    2559 scope.go:117] "RemoveContainer" containerID="2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8"
Jul 29 17:03:28.348094 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348057    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq4s6\" (UniqueName: \"kubernetes.io/projected/7b481619-8792-4c88-b866-73985a965137-kube-api-access-cq4s6\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348285 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348123    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b481619-8792-4c88-b866-73985a965137-tls-certs\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348285 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348182    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-home\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348285 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348217    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-tmp-dir\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348285 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348248    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-kserve-provision-location\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348512 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-model-cache\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348512 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-dshm\") pod \"7b481619-8792-4c88-b866-73985a965137\" (UID: \"7b481619-8792-4c88-b866-73985a965137\") "
Jul 29 17:03:28.348635 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.348602    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-home" (OuterVolumeSpecName: "home") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:28.349245 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.349008    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:28.349423 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.349404    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:28.351106 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.351064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b481619-8792-4c88-b866-73985a965137-kube-api-access-cq4s6" (OuterVolumeSpecName: "kube-api-access-cq4s6") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "kube-api-access-cq4s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:03:28.351771 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.351751    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-dshm" (OuterVolumeSpecName: "dshm") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:28.351903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.351880    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b481619-8792-4c88-b866-73985a965137-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:03:28.382210 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.382188    2559 scope.go:117] "RemoveContainer" containerID="4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0"
Jul 29 17:03:28.382602 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:03:28.382579    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0\": container with ID starting with 4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0 not found: ID does not exist" containerID="4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0"
Jul 29 17:03:28.382661 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.382612    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0"} err="failed to get container status \"4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0\": rpc error: code = NotFound desc = could not find container \"4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0\": container with ID starting with 4d835648bcc917cd2ee0826570735ab3cb57ada373c6c3d05f1b22d71cbe1ef0 not found: ID does not exist"
Jul 29 17:03:28.382661 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.382630    2559 scope.go:117] "RemoveContainer" containerID="2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8"
Jul 29 17:03:28.382958 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:03:28.382931    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8\": container with ID starting with 2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8 not found: ID does not exist" containerID="2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8"
Jul 29 17:03:28.383053 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.382964    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8"} err="failed to get container status \"2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8\": rpc error: code = NotFound desc = could not find container \"2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8\": container with ID starting with 2d1c6254b58f7fe05cc11af6088086e797a28a4a06065337451444bed2a42af8 not found: ID does not exist"
Jul 29 17:03:28.415165 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.415102    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b481619-8792-4c88-b866-73985a965137" (UID: "7b481619-8792-4c88-b866-73985a965137"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:03:28.449607 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449586    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b481619-8792-4c88-b866-73985a965137-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.449607 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449604    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-home\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.449701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449612    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-tmp-dir\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.449701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449622    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.449701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449632    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-model-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.449701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449640    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b481619-8792-4c88-b866-73985a965137-dshm\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.449701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.449648    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4s6\" (UniqueName: \"kubernetes.io/projected/7b481619-8792-4c88-b866-73985a965137-kube-api-access-cq4s6\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:03:28.636789 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.636762    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"]
Jul 29 17:03:28.642973 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.642942    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6749bdbf6-tblrg"]
Jul 29 17:03:28.705553 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:28.705517    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b481619-8792-4c88-b866-73985a965137" path="/var/lib/kubelet/pods/7b481619-8792-4c88-b866-73985a965137/volumes"
Jul 29 17:03:31.323017 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:31.322987    2559 generic.go:356] "Generic (PLEG): container finished" podID="f291298b-f689-4106-9459-f25a05cfde76" containerID="3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce" exitCode=0
Jul 29 17:03:31.323426 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:31.323050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" event={"ID":"f291298b-f689-4106-9459-f25a05cfde76","Type":"ContainerDied","Data":"3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce"}
Jul 29 17:03:32.328854 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:32.328821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" event={"ID":"f291298b-f689-4106-9459-f25a05cfde76","Type":"ContainerStarted","Data":"6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8"}
Jul 29 17:03:32.352982 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:32.352935    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" podStartSLOduration=2.216881644 podStartE2EDuration="6.352921491s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.136039847s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:03:32.350611217 +0000 UTC m=+730.200644916" watchObservedRunningTime="2026-07-29 17:03:32.352921491 +0000 UTC m=+730.202955189"
Jul 29 17:03:34.031126 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:34.031091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:34.096459 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:34.096410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:34.699230 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:34.699192    2559 scope.go:117] "RemoveContainer" containerID="0712c856e3c1d4cdd08e78c1ee9666caa4309294842ff2337b55d0443b566d00"
Jul 29 17:03:34.700274 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:34.700237    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:03:34.700385 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:34.700323    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:03:34.700385 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:34.700370    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 17:03:35.342233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:35.342198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" event={"ID":"581f680c-a2a0-4443-b3ca-4834dfaa9c17","Type":"ContainerStarted","Data":"6bd6fbba3e5f48c7a06496fea3ccb9b1d57f1014c7aa302c776be387038a5596"}
Jul 29 17:03:36.067798 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:36.067758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:03:36.068008 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:36.067886    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 29 17:03:36.393582 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:36.393507    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:36.393582 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:36.393553    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:36.406001 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:36.405973    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:37.066911 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:37.066864    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 29 17:03:37.374236 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:37.374138    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:03:38.066818 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:38.066772    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 29 17:03:39.066370 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:39.066325    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 29 17:03:40.066439 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:40.066391    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75" podUID="581f680c-a2a0-4443-b3ca-4834dfaa9c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 29 17:03:41.070774 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:41.070743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:03:41.071177 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:41.070997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:03:41.071687 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:41.071666    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-579df5d64f-t9b75"
Jul 29 17:03:43.067696 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:43.067658    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:43.339440 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:43.339363    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:03:53.253849 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:03:53.253812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:13.010190 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:13.010156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:13.064994 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:13.064963    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:20.009153 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:20.009108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:48.370608 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.370524    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"]
Jul 29 17:04:48.371023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.370787    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b481619-8792-4c88-b866-73985a965137" containerName="main"
Jul 29 17:04:48.371023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.370797    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b481619-8792-4c88-b866-73985a965137" containerName="main"
Jul 29 17:04:48.371023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.370812    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b481619-8792-4c88-b866-73985a965137" containerName="storage-initializer"
Jul 29 17:04:48.371023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.370818    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b481619-8792-4c88-b866-73985a965137" containerName="storage-initializer"
Jul 29 17:04:48.371023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.370867    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b481619-8792-4c88-b866-73985a965137" containerName="main"
Jul 29 17:04:48.372833 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.372817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.375525 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.375507    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 17:04:48.381759 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.381734    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"]
Jul 29 17:04:48.407236 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.407209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:48.460348 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.460432 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.460472 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-dshm\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.460511 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmnv\" (UniqueName: \"kubernetes.io/projected/017e152f-6e7b-4a01-a637-b28c969dfd13-kube-api-access-ggmnv\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.460511 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-model-cache\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.460511 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-home\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.460602 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.460535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017e152f-6e7b-4a01-a637-b28c969dfd13-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561072 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561204 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-dshm\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561204 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmnv\" (UniqueName: \"kubernetes.io/projected/017e152f-6e7b-4a01-a637-b28c969dfd13-kube-api-access-ggmnv\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561204 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-model-cache\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561204 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-home\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561204 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017e152f-6e7b-4a01-a637-b28c969dfd13-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561418 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561480 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561577 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-model-cache\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561637 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.561713 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.561689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-home\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.563429 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.563411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-dshm\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.563703 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.563684    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017e152f-6e7b-4a01-a637-b28c969dfd13-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.577147 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.577122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmnv\" (UniqueName: \"kubernetes.io/projected/017e152f-6e7b-4a01-a637-b28c969dfd13-kube-api-access-ggmnv\") pod \"scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.659215 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.659157    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"]
Jul 29 17:04:48.661406 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.661391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.663987 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.663968    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-wxllg\""
Jul 29 17:04:48.677135 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.677064    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"]
Jul 29 17:04:48.681315 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.681271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:48.682885 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.682868    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:48.762929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.762896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.763048 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.762944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5skw\" (UniqueName: \"kubernetes.io/projected/d6bc1573-bc3c-4493-8c71-b9016884c950-kube-api-access-q5skw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.763147 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.763121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.763199 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.763169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.763239 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.763211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.763281 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.763251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.810713 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.810687    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"]
Jul 29 17:04:48.813781 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:04:48.813750    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017e152f_6e7b_4a01_a637_b28c969dfd13.slice/crio-0310d7b071f9185fa85111370f96ee5966989ee9e3fa2c526b8934f096ddb0fb WatchSource:0}: Error finding container 0310d7b071f9185fa85111370f96ee5966989ee9e3fa2c526b8934f096ddb0fb: Status 404 returned error can't find the container with id 0310d7b071f9185fa85111370f96ee5966989ee9e3fa2c526b8934f096ddb0fb
Jul 29 17:04:48.863603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.863697 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863607    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.863697 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.863794 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.863854 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.863907 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5skw\" (UniqueName: \"kubernetes.io/projected/d6bc1573-bc3c-4493-8c71-b9016884c950-kube-api-access-q5skw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.863907 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863887    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.864001 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.864001 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.863984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.864153 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.864133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.866238 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.866216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.873994 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.873972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5skw\" (UniqueName: \"kubernetes.io/projected/d6bc1573-bc3c-4493-8c71-b9016884c950-kube-api-access-q5skw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:48.970647 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:48.970603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:04:49.116223 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:49.116183    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"]
Jul 29 17:04:49.123573 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:04:49.123487    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bc1573_bc3c_4493_8c71_b9016884c950.slice/crio-aa969fb7929886422525da170f699cdf9aa7425d119fa3239be4f9b3f4bb678a WatchSource:0}: Error finding container aa969fb7929886422525da170f699cdf9aa7425d119fa3239be4f9b3f4bb678a: Status 404 returned error can't find the container with id aa969fb7929886422525da170f699cdf9aa7425d119fa3239be4f9b3f4bb678a
Jul 29 17:04:49.621335 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:49.621294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" event={"ID":"017e152f-6e7b-4a01-a637-b28c969dfd13","Type":"ContainerStarted","Data":"efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496"}
Jul 29 17:04:49.621796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:49.621364    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" event={"ID":"017e152f-6e7b-4a01-a637-b28c969dfd13","Type":"ContainerStarted","Data":"0310d7b071f9185fa85111370f96ee5966989ee9e3fa2c526b8934f096ddb0fb"}
Jul 29 17:04:49.622916 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:49.622878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerStarted","Data":"14afc9679cb03757ff22945aa339c6bd7159fc48bed01823228ae4341c74bf08"}
Jul 29 17:04:49.623050 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:49.622928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerStarted","Data":"aa969fb7929886422525da170f699cdf9aa7425d119fa3239be4f9b3f4bb678a"}
Jul 29 17:04:50.092580 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:50.092543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:50.193455 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:50.193409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:50.628612 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:50.628529    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerID="14afc9679cb03757ff22945aa339c6bd7159fc48bed01823228ae4341c74bf08" exitCode=0
Jul 29 17:04:50.628953 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:50.628618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerDied","Data":"14afc9679cb03757ff22945aa339c6bd7159fc48bed01823228ae4341c74bf08"}
Jul 29 17:04:52.641737 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:52.641659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerStarted","Data":"a0a8405512b76e9f33b9ed21185333eb6b75e4ac14db1da20fc8f06fe4a63a8e"}
Jul 29 17:04:53.649045 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:53.648967    2559 generic.go:356] "Generic (PLEG): container finished" podID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerID="efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496" exitCode=0
Jul 29 17:04:53.649517 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:53.649100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" event={"ID":"017e152f-6e7b-4a01-a637-b28c969dfd13","Type":"ContainerDied","Data":"efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496"}
Jul 29 17:04:54.656381 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:54.656336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" event={"ID":"017e152f-6e7b-4a01-a637-b28c969dfd13","Type":"ContainerStarted","Data":"c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976"}
Jul 29 17:04:54.675181 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:54.675126    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" podStartSLOduration=2.133529006 podStartE2EDuration="6.675110869s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.541581863s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:04:54.674445159 +0000 UTC m=+812.524478859" watchObservedRunningTime="2026-07-29 17:04:54.675110869 +0000 UTC m=+812.525144565"
Jul 29 17:04:58.683729 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:58.683691    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:58.684185 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:58.683768    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:58.704467 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:58.704437    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:04:59.574670 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:59.574633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:04:59.693069 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:04:59.693042    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:05:20.772607 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:20.772515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerStarted","Data":"95038f55a192676776394fbc87c600bddc0fd1d7fa43125e49da87164626b75f"}
Jul 29 17:05:20.773067 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:20.772620    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:05:20.793230 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:20.793183    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podStartSLOduration=2.2604650680000002 podStartE2EDuration="32.793169595s" totalImagesPullingTime="29.825989744s" totalInitContainerRuntime="706.714783ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:04:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:05:20.791269877 +0000 UTC m=+838.641303576" watchObservedRunningTime="2026-07-29 17:05:20.793169595 +0000 UTC m=+838.643203293"
Jul 29 17:05:21.330679 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.330642    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"]
Jul 29 17:05:21.340474 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.340446    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"]
Jul 29 17:05:21.340740 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.340694    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerName="main" containerID="cri-o://c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976" gracePeriod=30
Jul 29 17:05:21.354916 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.354887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:21.446914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.446879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:21.589785 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.589729    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764197    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-home\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764256    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-model-cache\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764287    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-tmp-dir\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggmnv\" (UniqueName: \"kubernetes.io/projected/017e152f-6e7b-4a01-a637-b28c969dfd13-kube-api-access-ggmnv\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764408    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-kserve-provision-location\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-dshm\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017e152f-6e7b-4a01-a637-b28c969dfd13-tls-certs\") pod \"017e152f-6e7b-4a01-a637-b28c969dfd13\" (UID: \"017e152f-6e7b-4a01-a637-b28c969dfd13\") "
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764492    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-model-cache" (OuterVolumeSpecName: "model-cache") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764665    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-model-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.764956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.764928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:21.765591 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.765277    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-home" (OuterVolumeSpecName: "home") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:21.784095 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.783650    2559 generic.go:356] "Generic (PLEG): container finished" podID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerID="c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976" exitCode=0
Jul 29 17:05:21.784468 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.784103    2559 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" secret="" err="secret \"scheduler-ha-replicas-test-epp-sa-dockercfg-wxllg\" not found"
Jul 29 17:05:21.784468 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.784394    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"
Jul 29 17:05:21.786255 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.784715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" event={"ID":"017e152f-6e7b-4a01-a637-b28c969dfd13","Type":"ContainerDied","Data":"c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976"}
Jul 29 17:05:21.786255 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.784776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd" event={"ID":"017e152f-6e7b-4a01-a637-b28c969dfd13","Type":"ContainerDied","Data":"0310d7b071f9185fa85111370f96ee5966989ee9e3fa2c526b8934f096ddb0fb"}
Jul 29 17:05:21.786255 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.784805    2559 scope.go:117] "RemoveContainer" containerID="c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976"
Jul 29 17:05:21.786988 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.786643    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017e152f-6e7b-4a01-a637-b28c969dfd13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:05:21.786988 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.786795    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017e152f-6e7b-4a01-a637-b28c969dfd13-kube-api-access-ggmnv" (OuterVolumeSpecName: "kube-api-access-ggmnv") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "kube-api-access-ggmnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:05:21.789253 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.789220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-dshm" (OuterVolumeSpecName: "dshm") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:21.796106 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.796051    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 17:05:21.828269 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.828174    2559 scope.go:117] "RemoveContainer" containerID="efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496"
Jul 29 17:05:21.865490 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.865463    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017e152f-6e7b-4a01-a637-b28c969dfd13-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.865588 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.865493    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-home\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.865588 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.865506    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-tmp-dir\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.865588 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.865518    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmnv\" (UniqueName: \"kubernetes.io/projected/017e152f-6e7b-4a01-a637-b28c969dfd13-kube-api-access-ggmnv\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.865588 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.865531    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-dshm\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.872149 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.872125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "017e152f-6e7b-4a01-a637-b28c969dfd13" (UID: "017e152f-6e7b-4a01-a637-b28c969dfd13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:21.903301 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.903271    2559 scope.go:117] "RemoveContainer" containerID="c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976"
Jul 29 17:05:21.903647 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:21.903623    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976\": container with ID starting with c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976 not found: ID does not exist" containerID="c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976"
Jul 29 17:05:21.903713 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.903657    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976"} err="failed to get container status \"c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976\": rpc error: code = NotFound desc = could not find container \"c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976\": container with ID starting with c1f0974c0e59ac22bb71126764418820c76d13161dfaa4512d50c2a47c9ca976 not found: ID does not exist"
Jul 29 17:05:21.903713 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.903681    2559 scope.go:117] "RemoveContainer" containerID="efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496"
Jul 29 17:05:21.903954 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:21.903935    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496\": container with ID starting with efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496 not found: ID does not exist" containerID="efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496"
Jul 29 17:05:21.904013 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.903965    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496"} err="failed to get container status \"efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496\": rpc error: code = NotFound desc = could not find container \"efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496\": container with ID starting with efdb9e4b4637a3a12b8583b2ab01213dcf5701730d98164aefc79dc2312c6496 not found: ID does not exist"
Jul 29 17:05:21.966604 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:21.966570    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017e152f-6e7b-4a01-a637-b28c969dfd13-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:21.966743 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:21.966614    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:21.966743 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:21.966688    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs podName:d6bc1573-bc3c-4493-8c71-b9016884c950 nodeName:}" failed. No retries permitted until 2026-07-29 17:05:22.466668021 +0000 UTC m=+840.316701698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs") pod "scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:22.107467 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:22.107429    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"]
Jul 29 17:05:22.111482 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:22.111454    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-55bbc65c84-6qzjd"]
Jul 29 17:05:22.471448 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:22.471413    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:22.471613 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:22.471485    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs podName:d6bc1573-bc3c-4493-8c71-b9016884c950 nodeName:}" failed. No retries permitted until 2026-07-29 17:05:23.471467161 +0000 UTC m=+841.321500852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs") pod "scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:22.703725 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:22.703697    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" path="/var/lib/kubelet/pods/017e152f-6e7b-4a01-a637-b28c969dfd13/volumes"
Jul 29 17:05:22.787373 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:22.787277    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main" containerID="cri-o://a0a8405512b76e9f33b9ed21185333eb6b75e4ac14db1da20fc8f06fe4a63a8e" gracePeriod=30
Jul 29 17:05:22.787373 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:22.787328    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="tokenizer" containerID="cri-o://95038f55a192676776394fbc87c600bddc0fd1d7fa43125e49da87164626b75f" gracePeriod=30
Jul 29 17:05:22.788738 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:22.788711    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 17:05:23.479888 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:23.479858    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:23.480103 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:23.479937    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs podName:d6bc1573-bc3c-4493-8c71-b9016884c950 nodeName:}" failed. No retries permitted until 2026-07-29 17:05:25.479917339 +0000 UTC m=+843.329951015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs") pod "scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:23.798385 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:23.798248    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerID="a0a8405512b76e9f33b9ed21185333eb6b75e4ac14db1da20fc8f06fe4a63a8e" exitCode=0
Jul 29 17:05:23.798385 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:23.798321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerDied","Data":"a0a8405512b76e9f33b9ed21185333eb6b75e4ac14db1da20fc8f06fe4a63a8e"}
Jul 29 17:05:24.483834 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.483797    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"]
Jul 29 17:05:24.484157 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.484143    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerName="storage-initializer"
Jul 29 17:05:24.484211 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.484158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerName="storage-initializer"
Jul 29 17:05:24.484211 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.484173    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerName="main"
Jul 29 17:05:24.484211 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.484180    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerName="main"
Jul 29 17:05:24.484301 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.484229    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="017e152f-6e7b-4a01-a637-b28c969dfd13" containerName="main"
Jul 29 17:05:24.487328 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.487309    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.490044 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.490020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 29 17:05:24.498118 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.498073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"]
Jul 29 17:05:24.511548 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.511522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:24.590501 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.590630 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.590630 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590579    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-home\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.590700 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997pk\" (UniqueName: \"kubernetes.io/projected/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kube-api-access-997pk\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.590731 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.590793 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-dshm\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.590827 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.590796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-model-cache\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691494 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997pk\" (UniqueName: \"kubernetes.io/projected/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kube-api-access-997pk\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691787 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691787 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-dshm\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691787 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-model-cache\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691787 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691787 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.691787 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.691708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-home\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.692162 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.692058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-home\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.692525 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.692495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.692813 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.692788    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-model-cache\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.692959 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.692798    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.694837 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.694815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-dshm\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.695118 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.695099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.702018 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.701994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-997pk\" (UniqueName: \"kubernetes.io/projected/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kube-api-access-997pk\") pod \"precise-prefix-cache-test-kserve-66f68489b9-gd8b9\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.738886 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.738788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:24.797590 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.797554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:24.936997 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:24.936971    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"]
Jul 29 17:05:24.938943 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:05:24.938916    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37625eef_b57e_4e26_bdeb_8be13f7fa8e0.slice/crio-4a733a567312b75b3c7bb6a5d6d60b148d6883605f6542058105ecbfcf98149d WatchSource:0}: Error finding container 4a733a567312b75b3c7bb6a5d6d60b148d6883605f6542058105ecbfcf98149d: Status 404 returned error can't find the container with id 4a733a567312b75b3c7bb6a5d6d60b148d6883605f6542058105ecbfcf98149d
Jul 29 17:05:25.500945 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:25.500912    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:25.501122 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:25.500982    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs podName:d6bc1573-bc3c-4493-8c71-b9016884c950 nodeName:}" failed. No retries permitted until 2026-07-29 17:05:29.500967891 +0000 UTC m=+847.351001566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs") pod "scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:25.807799 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:25.807697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" event={"ID":"37625eef-b57e-4e26-bdeb-8be13f7fa8e0","Type":"ContainerStarted","Data":"f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc"}
Jul 29 17:05:25.807799 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:25.807768    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" event={"ID":"37625eef-b57e-4e26-bdeb-8be13f7fa8e0","Type":"ContainerStarted","Data":"4a733a567312b75b3c7bb6a5d6d60b148d6883605f6542058105ecbfcf98149d"}
Jul 29 17:05:28.970713 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:28.970676    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:05:29.539178 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:29.539113    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:29.539178 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:29.539177    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs podName:d6bc1573-bc3c-4493-8c71-b9016884c950 nodeName:}" failed. No retries permitted until 2026-07-29 17:05:37.539162741 +0000 UTC m=+855.389196421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs") pod "scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:29.824090 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:29.824008    2559 generic.go:356] "Generic (PLEG): container finished" podID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerID="f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc" exitCode=0
Jul 29 17:05:29.824221 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:29.824074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" event={"ID":"37625eef-b57e-4e26-bdeb-8be13f7fa8e0","Type":"ContainerDied","Data":"f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc"}
Jul 29 17:05:30.829539 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:30.829509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" event={"ID":"37625eef-b57e-4e26-bdeb-8be13f7fa8e0","Type":"ContainerStarted","Data":"031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa"}
Jul 29 17:05:30.849709 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:30.849667    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" podStartSLOduration=2.598606356 podStartE2EDuration="6.849651382s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.251045026s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:05:30.848626463 +0000 UTC m=+848.698660173" watchObservedRunningTime="2026-07-29 17:05:30.849651382 +0000 UTC m=+848.699685129"
Jul 29 17:05:32.788372 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:05:32.788346    2559 logging.go:55] [core] [Channel #23 SubChannel #24]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.27:9003", ServerName: "10.133.0.27:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.27:9003: connect: connection refused"
Jul 29 17:05:33.787974 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:33.787932    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.27:9003\" within 1s: context deadline exceeded"
Jul 29 17:05:34.797924 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:34.797887    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:34.798449 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:34.797949    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:34.810281 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:34.810254    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:34.856984 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:34.856957    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:34.980105 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:34.980056    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:37.608563 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:37.608531    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:37.608950 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:37.608607    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs podName:d6bc1573-bc3c-4493-8c71-b9016884c950 nodeName:}" failed. No retries permitted until 2026-07-29 17:05:53.608592875 +0000 UTC m=+871.458626551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs") pod "scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
Jul 29 17:05:42.788308 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:05:42.788277    2559 logging.go:55] [core] [Channel #25 SubChannel #26]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.27:9003", ServerName: "10.133.0.27:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.27:9003: connect: connection refused"
Jul 29 17:05:43.788385 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:43.788339    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.27:9003\" within 1s: context deadline exceeded"
Jul 29 17:05:52.787785 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:05:52.787758    2559 logging.go:55] [core] [Channel #27 SubChannel #28]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.27:9003", ServerName: "10.133.0.27:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.27:9003: connect: connection refused"
Jul 29 17:05:52.913946 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:52.913920    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww_d6bc1573-bc3c-4493-8c71-b9016884c950/tokenizer/0.log"
Jul 29 17:05:52.914613 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:52.914587    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerID="95038f55a192676776394fbc87c600bddc0fd1d7fa43125e49da87164626b75f" exitCode=137
Jul 29 17:05:52.914727 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:52.914632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerDied","Data":"95038f55a192676776394fbc87c600bddc0fd1d7fa43125e49da87164626b75f"}
Jul 29 17:05:53.463161 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.463135    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww_d6bc1573-bc3c-4493-8c71-b9016884c950/tokenizer/0.log"
Jul 29 17:05:53.478193 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.478166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:05:53.650021 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.649943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs\") pod \"d6bc1573-bc3c-4493-8c71-b9016884c950\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") "
Jul 29 17:05:53.650021 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.649996    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-uds\") pod \"d6bc1573-bc3c-4493-8c71-b9016884c950\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") "
Jul 29 17:05:53.650276 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650070    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-kserve-provision-location\") pod \"d6bc1573-bc3c-4493-8c71-b9016884c950\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") "
Jul 29 17:05:53.650276 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650141    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-tmp\") pod \"d6bc1573-bc3c-4493-8c71-b9016884c950\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") "
Jul 29 17:05:53.650276 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650170    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5skw\" (UniqueName: \"kubernetes.io/projected/d6bc1573-bc3c-4493-8c71-b9016884c950-kube-api-access-q5skw\") pod \"d6bc1573-bc3c-4493-8c71-b9016884c950\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") "
Jul 29 17:05:53.650276 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650210    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-cache\") pod \"d6bc1573-bc3c-4493-8c71-b9016884c950\" (UID: \"d6bc1573-bc3c-4493-8c71-b9016884c950\") "
Jul 29 17:05:53.650453 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650382    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d6bc1573-bc3c-4493-8c71-b9016884c950" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:53.650557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650504    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d6bc1573-bc3c-4493-8c71-b9016884c950" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:53.650557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650517    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d6bc1573-bc3c-4493-8c71-b9016884c950" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:53.650557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650517    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:53.650834 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.650810    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6bc1573-bc3c-4493-8c71-b9016884c950" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:53.652154 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.652131    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6bc1573-bc3c-4493-8c71-b9016884c950" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:05:53.652229 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.652182    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bc1573-bc3c-4493-8c71-b9016884c950-kube-api-access-q5skw" (OuterVolumeSpecName: "kube-api-access-q5skw") pod "d6bc1573-bc3c-4493-8c71-b9016884c950" (UID: "d6bc1573-bc3c-4493-8c71-b9016884c950"). InnerVolumeSpecName "kube-api-access-q5skw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:05:53.751822 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.751795    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:53.751822 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.751818    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:53.751959 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.751828    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5skw\" (UniqueName: \"kubernetes.io/projected/d6bc1573-bc3c-4493-8c71-b9016884c950-kube-api-access-q5skw\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:53.751959 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.751836    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bc1573-bc3c-4493-8c71-b9016884c950-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:53.751959 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.751845    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bc1573-bc3c-4493-8c71-b9016884c950-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:53.787929 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.787898    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.27:9003\" within 1s: context deadline exceeded"
Jul 29 17:05:53.788232 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:05:53.788006    2559 logging.go:55] [core] [Channel #27 SubChannel #28]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.27:9003", ServerName: "10.133.0.27:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.27:9003: operation was canceled"
Jul 29 17:05:53.919756 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.919690    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww_d6bc1573-bc3c-4493-8c71-b9016884c950/tokenizer/0.log"
Jul 29 17:05:53.920337 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.920316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww" event={"ID":"d6bc1573-bc3c-4493-8c71-b9016884c950","Type":"ContainerDied","Data":"aa969fb7929886422525da170f699cdf9aa7425d119fa3239be4f9b3f4bb678a"}
Jul 29 17:05:53.920421 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.920338    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"
Jul 29 17:05:53.920421 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.920372    2559 scope.go:117] "RemoveContainer" containerID="95038f55a192676776394fbc87c600bddc0fd1d7fa43125e49da87164626b75f"
Jul 29 17:05:53.928329 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.928306    2559 scope.go:117] "RemoveContainer" containerID="a0a8405512b76e9f33b9ed21185333eb6b75e4ac14db1da20fc8f06fe4a63a8e"
Jul 29 17:05:53.935668 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.935650    2559 scope.go:117] "RemoveContainer" containerID="14afc9679cb03757ff22945aa339c6bd7159fc48bed01823228ae4341c74bf08"
Jul 29 17:05:53.943411 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.943393    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"]
Jul 29 17:05:53.946158 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:53.946138    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-74c9c5fvnkww"]
Jul 29 17:05:54.709290 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:54.709257    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" path="/var/lib/kubelet/pods/d6bc1573-bc3c-4493-8c71-b9016884c950/volumes"
Jul 29 17:05:57.968382 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:57.968348    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"]
Jul 29 17:05:57.968975 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:57.968582    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerName="main" containerID="cri-o://031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa" gracePeriod=30
Jul 29 17:05:57.978968 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:57.978941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:58.080232 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.080197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:05:58.219873 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.219818    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:58.390392 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390343    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-home\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390583 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-997pk\" (UniqueName: \"kubernetes.io/projected/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kube-api-access-997pk\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390583 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390450    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tmp-dir\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390583 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kserve-provision-location\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390756 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390674    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-home" (OuterVolumeSpecName: "home") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.390756 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tls-certs\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390756 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390700    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.390756 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-model-cache\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390933 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390779    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-dshm\") pod \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\" (UID: \"37625eef-b57e-4e26-bdeb-8be13f7fa8e0\") "
Jul 29 17:05:58.390968 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.390943    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.391103 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.391068    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-model-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.391166 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.391111    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-home\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.391166 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.391124    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tmp-dir\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.392797 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.392776    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-dshm" (OuterVolumeSpecName: "dshm") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.392870 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.392804    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:05:58.392962 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.392941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kube-api-access-997pk" (OuterVolumeSpecName: "kube-api-access-997pk") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "kube-api-access-997pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:05:58.446343 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.446286    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37625eef-b57e-4e26-bdeb-8be13f7fa8e0" (UID: "37625eef-b57e-4e26-bdeb-8be13f7fa8e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:05:58.492448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.492390    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.492448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.492414    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.492448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.492424    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-dshm\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.492448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.492432    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997pk\" (UniqueName: \"kubernetes.io/projected/37625eef-b57e-4e26-bdeb-8be13f7fa8e0-kube-api-access-997pk\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:05:58.944031 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.944000    2559 generic.go:356] "Generic (PLEG): container finished" podID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerID="031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa" exitCode=0
Jul 29 17:05:58.944250 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.944050    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"
Jul 29 17:05:58.944250 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.944093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" event={"ID":"37625eef-b57e-4e26-bdeb-8be13f7fa8e0","Type":"ContainerDied","Data":"031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa"}
Jul 29 17:05:58.944250 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.944156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9" event={"ID":"37625eef-b57e-4e26-bdeb-8be13f7fa8e0","Type":"ContainerDied","Data":"4a733a567312b75b3c7bb6a5d6d60b148d6883605f6542058105ecbfcf98149d"}
Jul 29 17:05:58.944250 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.944186    2559 scope.go:117] "RemoveContainer" containerID="031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa"
Jul 29 17:05:58.952188 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.952161    2559 scope.go:117] "RemoveContainer" containerID="f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc"
Jul 29 17:05:58.963763 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.963737    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"]
Jul 29 17:05:58.968076 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:58.968054    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-66f68489b9-gd8b9"]
Jul 29 17:05:59.012807 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:59.012660    2559 scope.go:117] "RemoveContainer" containerID="031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa"
Jul 29 17:05:59.013005 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:59.012967    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa\": container with ID starting with 031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa not found: ID does not exist" containerID="031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa"
Jul 29 17:05:59.013005 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:59.012988    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa"} err="failed to get container status \"031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa\": rpc error: code = NotFound desc = could not find container \"031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa\": container with ID starting with 031cebd1a95d85ec09a0d8449ac95ab2efb27eac6a37acb9dd086e20fec997aa not found: ID does not exist"
Jul 29 17:05:59.013097 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:59.013005    2559 scope.go:117] "RemoveContainer" containerID="f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc"
Jul 29 17:05:59.013325 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:05:59.013293    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc\": container with ID starting with f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc not found: ID does not exist" containerID="f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc"
Jul 29 17:05:59.013432 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:05:59.013327    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc"} err="failed to get container status \"f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc\": rpc error: code = NotFound desc = could not find container \"f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc\": container with ID starting with f3aac2ab593a94a9a18678542e6093d94133d70ac23bcb21818b7b08712950cc not found: ID does not exist"
Jul 29 17:06:00.707514 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:06:00.707481    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" path="/var/lib/kubelet/pods/37625eef-b57e-4e26-bdeb-8be13f7fa8e0/volumes"
Jul 29 17:06:12.109177 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:06:12.109140    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:06:12.301120 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:06:12.301076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:06:37.604196 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:06:37.604152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:07:49.672337 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:49.672254    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"]
Jul 29 17:07:49.672822 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:49.672469    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" podUID="f291298b-f689-4106-9459-f25a05cfde76" containerName="main" containerID="cri-o://6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8" gracePeriod=30
Jul 29 17:07:49.693597 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:49.693558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:07:49.916115 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:49.916070    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:07:50.017023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.016995    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmfvh\" (UniqueName: \"kubernetes.io/projected/f291298b-f689-4106-9459-f25a05cfde76-kube-api-access-wmfvh\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-dshm\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017246 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-kserve-provision-location\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017246 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f291298b-f689-4106-9459-f25a05cfde76-tls-certs\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017246 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017185    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-home\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017246 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-tmp-dir\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017456 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017255    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-model-cache\") pod \"f291298b-f689-4106-9459-f25a05cfde76\" (UID: \"f291298b-f689-4106-9459-f25a05cfde76\") "
Jul 29 17:07:50.017521 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017486    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-home" (OuterVolumeSpecName: "home") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:07:50.017521 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017490    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:07:50.017669 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.017641    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-model-cache" (OuterVolumeSpecName: "model-cache") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:07:50.019151 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.019129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f291298b-f689-4106-9459-f25a05cfde76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:07:50.019546 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.019526    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-dshm" (OuterVolumeSpecName: "dshm") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:07:50.019613 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.019547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f291298b-f689-4106-9459-f25a05cfde76-kube-api-access-wmfvh" (OuterVolumeSpecName: "kube-api-access-wmfvh") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "kube-api-access-wmfvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:07:50.079015 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.078989    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f291298b-f689-4106-9459-f25a05cfde76" (UID: "f291298b-f689-4106-9459-f25a05cfde76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:07:50.118754 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118734    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.118754 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118753    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f291298b-f689-4106-9459-f25a05cfde76-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.118871 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118764    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-home\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.118871 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118771    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-tmp-dir\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.118871 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118779    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-model-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.118871 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118787    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmfvh\" (UniqueName: \"kubernetes.io/projected/f291298b-f689-4106-9459-f25a05cfde76-kube-api-access-wmfvh\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.118871 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.118793    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f291298b-f689-4106-9459-f25a05cfde76-dshm\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:07:50.350974 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.350898    2559 generic.go:356] "Generic (PLEG): container finished" podID="f291298b-f689-4106-9459-f25a05cfde76" containerID="6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8" exitCode=0
Jul 29 17:07:50.350974 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.350952    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"
Jul 29 17:07:50.351146 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.350978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" event={"ID":"f291298b-f689-4106-9459-f25a05cfde76","Type":"ContainerDied","Data":"6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8"}
Jul 29 17:07:50.351146 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.351032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw" event={"ID":"f291298b-f689-4106-9459-f25a05cfde76","Type":"ContainerDied","Data":"ec92d0cfcf36c249759508c6dd2320a0e80cae1b0e7dacc867fa92614ee4919e"}
Jul 29 17:07:50.351146 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.351055    2559 scope.go:117] "RemoveContainer" containerID="6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8"
Jul 29 17:07:50.359780 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.359764    2559 scope.go:117] "RemoveContainer" containerID="3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce"
Jul 29 17:07:50.369879 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.369836    2559 scope.go:117] "RemoveContainer" containerID="6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8"
Jul 29 17:07:50.370222 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:07:50.370196    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8\": container with ID starting with 6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8 not found: ID does not exist" containerID="6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8"
Jul 29 17:07:50.370306 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.370223    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8"} err="failed to get container status \"6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8\": rpc error: code = NotFound desc = could not find container \"6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8\": container with ID starting with 6ecd5ec56573dc6d19408cdf0169530aee42531c0eaaeb611f1c5f0d84a011b8 not found: ID does not exist"
Jul 29 17:07:50.370306 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.370249    2559 scope.go:117] "RemoveContainer" containerID="3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce"
Jul 29 17:07:50.370603 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:07:50.370588    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce\": container with ID starting with 3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce not found: ID does not exist" containerID="3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce"
Jul 29 17:07:50.370654 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.370609    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce"} err="failed to get container status \"3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce\": rpc error: code = NotFound desc = could not find container \"3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce\": container with ID starting with 3f0c31cf123f23423d5cd4b887a53c4ce66302da2936627e8c25bdcaa93258ce not found: ID does not exist"
Jul 29 17:07:50.371809 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.371788    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"]
Jul 29 17:07:50.379767 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.379738    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6fd74f44dd9zsjw"]
Jul 29 17:07:50.703239 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:07:50.703212    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f291298b-f689-4106-9459-f25a05cfde76" path="/var/lib/kubelet/pods/f291298b-f689-4106-9459-f25a05cfde76/volumes"
Jul 29 17:08:04.415966 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.415927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:08:04.754999 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.754967    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"]
Jul 29 17:08:04.755267 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755254    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main"
Jul 29 17:08:04.755311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755269    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main"
Jul 29 17:08:04.755311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755283    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f291298b-f689-4106-9459-f25a05cfde76" containerName="storage-initializer"
Jul 29 17:08:04.755311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755289    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f291298b-f689-4106-9459-f25a05cfde76" containerName="storage-initializer"
Jul 29 17:08:04.755311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755299    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="tokenizer"
Jul 29 17:08:04.755311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755304    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="tokenizer"
Jul 29 17:08:04.755311 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755311    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerName="storage-initializer"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755316    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerName="storage-initializer"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755324    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755329    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755337    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f291298b-f689-4106-9459-f25a05cfde76" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755342    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f291298b-f689-4106-9459-f25a05cfde76" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755350    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="storage-initializer"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755356    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="storage-initializer"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755401    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f291298b-f689-4106-9459-f25a05cfde76" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755408    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37625eef-b57e-4e26-bdeb-8be13f7fa8e0" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755414    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="main"
Jul 29 17:08:04.755487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.755419    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6bc1573-bc3c-4493-8c71-b9016884c950" containerName="tokenizer"
Jul 29 17:08:04.758367 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.758353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.760973 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.760950    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:08:04.761908 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.761889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-hq4fq\""
Jul 29 17:08:04.761994 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.761889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Jul 29 17:08:04.771066 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.771041    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"]
Jul 29 17:08:04.827138 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.827099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7clj\" (UniqueName: \"kubernetes.io/projected/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kube-api-access-n7clj\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.827270 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.827156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.827270 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.827204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.827270 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.827236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.827270 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.827263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.827453 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.827295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.854959 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.854916    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:08:04.927701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.927665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.927903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.927706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.927903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.927734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.927903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.927766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7clj\" (UniqueName: \"kubernetes.io/projected/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kube-api-access-n7clj\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.927903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.927789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.927903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.927821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.928183 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.928156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.928232 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.928158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.928276 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.928234    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.928276 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.928256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.930458 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.930436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:04.947075 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:04.947040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7clj\" (UniqueName: \"kubernetes.io/projected/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kube-api-access-n7clj\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:05.066867 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:05.066785    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:05.200959 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:05.200883    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"]
Jul 29 17:08:05.203628 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:08:05.203598    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b02589_3597_4b2c_bfc3_fc6e9ecfb02d.slice/crio-fea38b1d64522d3765667eb6b736dcaa387b91039fc14ea74b52dd29250812dd WatchSource:0}: Error finding container fea38b1d64522d3765667eb6b736dcaa387b91039fc14ea74b52dd29250812dd: Status 404 returned error can't find the container with id fea38b1d64522d3765667eb6b736dcaa387b91039fc14ea74b52dd29250812dd
Jul 29 17:08:05.409461 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:05.409373    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerStarted","Data":"fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3"}
Jul 29 17:08:05.409461 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:05.409452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerStarted","Data":"fea38b1d64522d3765667eb6b736dcaa387b91039fc14ea74b52dd29250812dd"}
Jul 29 17:08:06.415354 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:06.415323    2559 generic.go:356] "Generic (PLEG): container finished" podID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerID="fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3" exitCode=0
Jul 29 17:08:06.415785 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:06.415412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerDied","Data":"fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3"}
Jul 29 17:08:07.421941 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:07.421902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerStarted","Data":"230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58"}
Jul 29 17:08:07.422350 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:07.421959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerStarted","Data":"5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4"}
Jul 29 17:08:07.422350 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:07.422108    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:07.445290 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:07.445241    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" podStartSLOduration=2.776472789 podStartE2EDuration="3.445226931s" totalImagesPullingTime="0s" totalInitContainerRuntime="668.754142ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:08:07.44349003 +0000 UTC m=+1005.293523731" watchObservedRunningTime="2026-07-29 17:08:07.445226931 +0000 UTC m=+1005.295260628"
Jul 29 17:08:15.067358 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:15.067324    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:15.067801 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:15.067387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:15.069897 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:15.069877    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:15.454890 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:15.454857    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:16.060051 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:16.060018    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:08:25.685622 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:25.685573    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:08:36.459402 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:36.459374    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:08:53.980281 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:08:53.980229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:00.180799 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:00.180759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:00.247482 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:00.247450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:09.595435 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.595403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:09.747232 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.747200    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"]
Jul 29 17:09:09.749952 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.749936    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.752513 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.752492    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 17:09:09.752513 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.752495    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-2vzkl\""
Jul 29 17:09:09.761775 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.761752    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"]
Jul 29 17:09:09.774292 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.774265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:09.827350 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.827325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2d584d-b180-4617-9cb4-ae917484f112-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.827463 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.827368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9m7n\" (UniqueName: \"kubernetes.io/projected/de2d584d-b180-4617-9cb4-ae917484f112-kube-api-access-h9m7n\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.827463 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.827397    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.827463 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.827434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.827567 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.827464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.827567 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.827481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928504 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928504 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928602 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928653 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2d584d-b180-4617-9cb4-ae917484f112-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928715 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9m7n\" (UniqueName: \"kubernetes.io/projected/de2d584d-b180-4617-9cb4-ae917484f112-kube-api-access-h9m7n\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928792 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928792 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928905 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.928905 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.928885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.930971 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.930946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2d584d-b180-4617-9cb4-ae917484f112-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:09.936779 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:09.936752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9m7n\" (UniqueName: \"kubernetes.io/projected/de2d584d-b180-4617-9cb4-ae917484f112-kube-api-access-h9m7n\") pod \"stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:10.059061 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:10.059030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:10.185236 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:10.185212    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"]
Jul 29 17:09:10.187242 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:09:10.187212    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2d584d_b180_4617_9cb4_ae917484f112.slice/crio-4bc50af82a52dcc9ad068943ad7309e0f9a65831894c6b61ca83205f69ab2fd9 WatchSource:0}: Error finding container 4bc50af82a52dcc9ad068943ad7309e0f9a65831894c6b61ca83205f69ab2fd9: Status 404 returned error can't find the container with id 4bc50af82a52dcc9ad068943ad7309e0f9a65831894c6b61ca83205f69ab2fd9
Jul 29 17:09:10.655642 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:10.655561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerStarted","Data":"6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d"}
Jul 29 17:09:10.656073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:10.656031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerStarted","Data":"4bc50af82a52dcc9ad068943ad7309e0f9a65831894c6b61ca83205f69ab2fd9"}
Jul 29 17:09:11.660515 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:11.660480    2559 generic.go:356] "Generic (PLEG): container finished" podID="de2d584d-b180-4617-9cb4-ae917484f112" containerID="6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d" exitCode=0
Jul 29 17:09:11.660918 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:11.660556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerDied","Data":"6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d"}
Jul 29 17:09:12.667333 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:12.667296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerStarted","Data":"c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e"}
Jul 29 17:09:12.667722 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:12.667401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerStarted","Data":"cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf"}
Jul 29 17:09:12.667722 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:12.667438    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:12.692295 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:12.692251    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" podStartSLOduration=3.045138917 podStartE2EDuration="3.692235687s" totalImagesPullingTime="0s" totalInitContainerRuntime="647.09677ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:09:12.690845842 +0000 UTC m=+1070.540879553" watchObservedRunningTime="2026-07-29 17:09:12.692235687 +0000 UTC m=+1070.542269384"
Jul 29 17:09:13.976110 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:13.976053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:15.775559 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:15.775525    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:09:20.059489 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:20.059458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:20.059943 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:20.059515    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:20.062345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:20.062324    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:20.703561 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:20.703535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:09:41.704316 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:09:41.704288    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:10:01.969562 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:01.969523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:01.977244 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:01.977217    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"]
Jul 29 17:10:01.977475 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:01.977450    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="main" containerID="cri-o://5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4" gracePeriod=30
Jul 29 17:10:01.977549 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:01.977497    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="tokenizer" containerID="cri-o://230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58" gracePeriod=30
Jul 29 17:10:02.141678 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:02.141639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:02.859195 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:02.859160    2559 generic.go:356] "Generic (PLEG): container finished" podID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerID="5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4" exitCode=0
Jul 29 17:10:02.859365 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:02.859198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerDied","Data":"5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4"}
Jul 29 17:10:03.122416 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.122361    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:10:03.161182 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161158    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7clj\" (UniqueName: \"kubernetes.io/projected/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kube-api-access-n7clj\") pod \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") "
Jul 29 17:10:03.161352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161188    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-cache\") pod \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") "
Jul 29 17:10:03.161352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161206    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kserve-provision-location\") pod \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") "
Jul 29 17:10:03.161352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161232    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-tmp\") pod \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") "
Jul 29 17:10:03.161352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161277    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tls-certs\") pod \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") "
Jul 29 17:10:03.161352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161348    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-uds\") pod \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\" (UID: \"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d\") "
Jul 29 17:10:03.161628 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161461    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" (UID: "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:03.161628 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161576    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:10:03.161729 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161660    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" (UID: "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:03.161729 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.161672    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" (UID: "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:03.162047 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.162024    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" (UID: "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:10:03.163313 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.163288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kube-api-access-n7clj" (OuterVolumeSpecName: "kube-api-access-n7clj") pod "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" (UID: "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d"). InnerVolumeSpecName "kube-api-access-n7clj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:10:03.163615 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.163595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" (UID: "f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:10:03.262388 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.262341    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:10:03.262388 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.262382    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7clj\" (UniqueName: \"kubernetes.io/projected/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kube-api-access-n7clj\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:10:03.262388 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.262393    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:10:03.262388 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.262402    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:10:03.262652 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.262412    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:10:03.864710 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.864676    2559 generic.go:356] "Generic (PLEG): container finished" podID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerID="230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58" exitCode=0
Jul 29 17:10:03.864918 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.864739    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"
Jul 29 17:10:03.864918 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.864765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerDied","Data":"230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58"}
Jul 29 17:10:03.864918 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.864830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc" event={"ID":"f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d","Type":"ContainerDied","Data":"fea38b1d64522d3765667eb6b736dcaa387b91039fc14ea74b52dd29250812dd"}
Jul 29 17:10:03.864918 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.864863    2559 scope.go:117] "RemoveContainer" containerID="230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58"
Jul 29 17:10:03.873112 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.873072    2559 scope.go:117] "RemoveContainer" containerID="5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4"
Jul 29 17:10:03.880027 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.880011    2559 scope.go:117] "RemoveContainer" containerID="fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3"
Jul 29 17:10:03.885835 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.885815    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"]
Jul 29 17:10:03.887055 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.887044    2559 scope.go:117] "RemoveContainer" containerID="230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58"
Jul 29 17:10:03.887298 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:10:03.887284    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58\": container with ID starting with 230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58 not found: ID does not exist" containerID="230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58"
Jul 29 17:10:03.887348 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.887305    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58"} err="failed to get container status \"230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58\": rpc error: code = NotFound desc = could not find container \"230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58\": container with ID starting with 230821b6a0844b3c3bdeb4dbe323293bd16935083418141a45be13c8810e2f58 not found: ID does not exist"
Jul 29 17:10:03.887348 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.887321    2559 scope.go:117] "RemoveContainer" containerID="5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4"
Jul 29 17:10:03.887554 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:10:03.887537    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4\": container with ID starting with 5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4 not found: ID does not exist" containerID="5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4"
Jul 29 17:10:03.887595 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.887560    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4"} err="failed to get container status \"5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4\": rpc error: code = NotFound desc = could not find container \"5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4\": container with ID starting with 5cfdd3d4a4294c39dfda2f4e0a599d7650bb55a659014231c40e738896d7fde4 not found: ID does not exist"
Jul 29 17:10:03.887595 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.887576    2559 scope.go:117] "RemoveContainer" containerID="fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3"
Jul 29 17:10:03.887786 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:10:03.887771    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3\": container with ID starting with fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3 not found: ID does not exist" containerID="fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3"
Jul 29 17:10:03.887829 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.887789    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3"} err="failed to get container status \"fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3\": rpc error: code = NotFound desc = could not find container \"fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3\": container with ID starting with fcb167069f5bf70c8ef9310aec25c5d2e7c909ef1acb81d57787897366d894e3 not found: ID does not exist"
Jul 29 17:10:03.890488 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:03.890467    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqwhzc"]
Jul 29 17:10:04.703348 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:04.703316    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" path="/var/lib/kubelet/pods/f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d/volumes"
Jul 29 17:10:10.597854 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.597818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:10.803976 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.803940    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"]
Jul 29 17:10:10.804241 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804227    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="main"
Jul 29 17:10:10.804294 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804242    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="main"
Jul 29 17:10:10.804294 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804254    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="storage-initializer"
Jul 29 17:10:10.804294 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804261    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="storage-initializer"
Jul 29 17:10:10.804294 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804270    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="tokenizer"
Jul 29 17:10:10.804294 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804276    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="tokenizer"
Jul 29 17:10:10.804448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804326    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="tokenizer"
Jul 29 17:10:10.804448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.804334    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b02589-3597-4b2c-bfc3-fc6e9ecfb02d" containerName="main"
Jul 29 17:10:10.808896 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.808878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:10.811670 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.811645    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-n7kc6\""
Jul 29 17:10:10.811781 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.811655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 29 17:10:10.820868 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.820846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"]
Jul 29 17:10:10.835824 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.835799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:10.924005 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.923927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdz2\" (UniqueName: \"kubernetes.io/projected/4aed0aac-a5a9-4348-b812-766c16e6d40a-kube-api-access-2wdz2\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:10.924005 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.923965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:10.924005 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.923984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:10.924321 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.924164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:10.924321 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.924217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed0aac-a5a9-4348-b812-766c16e6d40a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:10.924321 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:10.924247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025157 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025340 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed0aac-a5a9-4348-b812-766c16e6d40a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025340 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025340 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdz2\" (UniqueName: \"kubernetes.io/projected/4aed0aac-a5a9-4348-b812-766c16e6d40a-kube-api-access-2wdz2\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025496 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025496 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025612 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025671 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025747 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.025804 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.025780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.027727 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.027702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed0aac-a5a9-4348-b812-766c16e6d40a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.034360 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.034330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdz2\" (UniqueName: \"kubernetes.io/projected/4aed0aac-a5a9-4348-b812-766c16e6d40a-kube-api-access-2wdz2\") pod \"custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.118023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.117977    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:11.248060 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.248037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"]
Jul 29 17:10:11.250391 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:10:11.250363    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aed0aac_a5a9_4348_b812_766c16e6d40a.slice/crio-ba28d5aaa00b5b227298008428f0607711ea5499950b37774782784d419718d4 WatchSource:0}: Error finding container ba28d5aaa00b5b227298008428f0607711ea5499950b37774782784d419718d4: Status 404 returned error can't find the container with id ba28d5aaa00b5b227298008428f0607711ea5499950b37774782784d419718d4
Jul 29 17:10:11.898357 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.898325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerStarted","Data":"1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984"}
Jul 29 17:10:11.898761 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:11.898375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerStarted","Data":"ba28d5aaa00b5b227298008428f0607711ea5499950b37774782784d419718d4"}
Jul 29 17:10:12.903330 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:12.903294    2559 generic.go:356] "Generic (PLEG): container finished" podID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerID="1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984" exitCode=0
Jul 29 17:10:12.903868 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:12.903373    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerDied","Data":"1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984"}
Jul 29 17:10:13.914651 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:13.914616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerStarted","Data":"62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd"}
Jul 29 17:10:13.915021 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:13.914712    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:13.915021 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:13.914731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerStarted","Data":"fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7"}
Jul 29 17:10:13.936236 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:13.936184    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" podStartSLOduration=3.26663169 podStartE2EDuration="3.936169673s" totalImagesPullingTime="0s" totalInitContainerRuntime="669.537983ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:10:13.93394751 +0000 UTC m=+1131.783981207" watchObservedRunningTime="2026-07-29 17:10:13.936169673 +0000 UTC m=+1131.786203372"
Jul 29 17:10:21.118870 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:21.118811    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:21.119389 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:21.118938    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:21.121701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:21.121680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:21.946374 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:21.946337    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:33.380758 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:33.380711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:42.075731 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:42.075691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:42.132877 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:42.132841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:42.951139 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:42.951111    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:10:44.872304 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:44.872202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:10:44.927488 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:10:44.927456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:11:01.147471 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:01.147390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:11:01.152656 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:01.152631    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"]
Jul 29 17:11:01.152930 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:01.152907    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="main" containerID="cri-o://cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf" gracePeriod=30
Jul 29 17:11:01.153017 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:01.152941    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="tokenizer" containerID="cri-o://c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e" gracePeriod=30
Jul 29 17:11:01.248453 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:01.248416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:11:01.703524 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:11:01.703492    2559 logging.go:55] [core] [Channel #167 SubChannel #168]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.30:9003", ServerName: "10.133.0.30:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.30:9003: connect: connection refused"
Jul 29 17:11:02.094442 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.094416    2559 generic.go:356] "Generic (PLEG): container finished" podID="de2d584d-b180-4617-9cb4-ae917484f112" containerID="cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf" exitCode=0
Jul 29 17:11:02.094570 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.094490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerDied","Data":"cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf"}
Jul 29 17:11:02.300305 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.300280    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:11:02.329345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.328579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-cache\") pod \"de2d584d-b180-4617-9cb4-ae917484f112\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") "
Jul 29 17:11:02.329345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.328643    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-uds\") pod \"de2d584d-b180-4617-9cb4-ae917484f112\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") "
Jul 29 17:11:02.329345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.328682    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2d584d-b180-4617-9cb4-ae917484f112-tls-certs\") pod \"de2d584d-b180-4617-9cb4-ae917484f112\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") "
Jul 29 17:11:02.329345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.328731    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-tmp\") pod \"de2d584d-b180-4617-9cb4-ae917484f112\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") "
Jul 29 17:11:02.329345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.328796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-kserve-provision-location\") pod \"de2d584d-b180-4617-9cb4-ae917484f112\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") "
Jul 29 17:11:02.329345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.328830    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9m7n\" (UniqueName: \"kubernetes.io/projected/de2d584d-b180-4617-9cb4-ae917484f112-kube-api-access-h9m7n\") pod \"de2d584d-b180-4617-9cb4-ae917484f112\" (UID: \"de2d584d-b180-4617-9cb4-ae917484f112\") "
Jul 29 17:11:02.329724 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.329390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "de2d584d-b180-4617-9cb4-ae917484f112" (UID: "de2d584d-b180-4617-9cb4-ae917484f112"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:02.329724 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.329417    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "de2d584d-b180-4617-9cb4-ae917484f112" (UID: "de2d584d-b180-4617-9cb4-ae917484f112"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:02.329724 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.329575    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "de2d584d-b180-4617-9cb4-ae917484f112" (UID: "de2d584d-b180-4617-9cb4-ae917484f112"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:02.330429 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.330403    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de2d584d-b180-4617-9cb4-ae917484f112" (UID: "de2d584d-b180-4617-9cb4-ae917484f112"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:11:02.331437 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.331416    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2d584d-b180-4617-9cb4-ae917484f112-kube-api-access-h9m7n" (OuterVolumeSpecName: "kube-api-access-h9m7n") pod "de2d584d-b180-4617-9cb4-ae917484f112" (UID: "de2d584d-b180-4617-9cb4-ae917484f112"). InnerVolumeSpecName "kube-api-access-h9m7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:11:02.331606 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.331586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2d584d-b180-4617-9cb4-ae917484f112-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de2d584d-b180-4617-9cb4-ae917484f112" (UID: "de2d584d-b180-4617-9cb4-ae917484f112"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:11:02.429845 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.429815    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:11:02.429845 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.429843    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:11:02.430035 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.429855    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2d584d-b180-4617-9cb4-ae917484f112-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:11:02.430035 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.429867    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:11:02.430035 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.429878    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2d584d-b180-4617-9cb4-ae917484f112-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:11:02.430035 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.429890    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9m7n\" (UniqueName: \"kubernetes.io/projected/de2d584d-b180-4617-9cb4-ae917484f112-kube-api-access-h9m7n\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:11:02.703275 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:02.703233    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.30:9003\" within 1s: context deadline exceeded"
Jul 29 17:11:03.099500 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.099411    2559 generic.go:356] "Generic (PLEG): container finished" podID="de2d584d-b180-4617-9cb4-ae917484f112" containerID="c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e" exitCode=0
Jul 29 17:11:03.099500 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.099476    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"
Jul 29 17:11:03.099700 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.099493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerDied","Data":"c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e"}
Jul 29 17:11:03.099700 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.099549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88" event={"ID":"de2d584d-b180-4617-9cb4-ae917484f112","Type":"ContainerDied","Data":"4bc50af82a52dcc9ad068943ad7309e0f9a65831894c6b61ca83205f69ab2fd9"}
Jul 29 17:11:03.099700 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.099573    2559 scope.go:117] "RemoveContainer" containerID="c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e"
Jul 29 17:11:03.107480 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.107462    2559 scope.go:117] "RemoveContainer" containerID="cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf"
Jul 29 17:11:03.116414 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.116387    2559 scope.go:117] "RemoveContainer" containerID="6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d"
Jul 29 17:11:03.118213 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.118191    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"]
Jul 29 17:11:03.123423 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.123397    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79bf4574b-2ph88"]
Jul 29 17:11:03.124021 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.124006    2559 scope.go:117] "RemoveContainer" containerID="c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e"
Jul 29 17:11:03.124291 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:11:03.124273    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e\": container with ID starting with c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e not found: ID does not exist" containerID="c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e"
Jul 29 17:11:03.124345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.124300    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e"} err="failed to get container status \"c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e\": rpc error: code = NotFound desc = could not find container \"c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e\": container with ID starting with c199439d55f4aa27d57553f499d59743a756a0b7f45380736e2d4d10ca8cd28e not found: ID does not exist"
Jul 29 17:11:03.124345 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.124317    2559 scope.go:117] "RemoveContainer" containerID="cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf"
Jul 29 17:11:03.124523 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:11:03.124506    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf\": container with ID starting with cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf not found: ID does not exist" containerID="cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf"
Jul 29 17:11:03.124571 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.124526    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf"} err="failed to get container status \"cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf\": rpc error: code = NotFound desc = could not find container \"cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf\": container with ID starting with cde4637c9af2a6971b70aeb2d68d1131cf4b2e95a3f5d39709f47d57e52b9fcf not found: ID does not exist"
Jul 29 17:11:03.124571 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.124540    2559 scope.go:117] "RemoveContainer" containerID="6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d"
Jul 29 17:11:03.124757 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:11:03.124734    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d\": container with ID starting with 6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d not found: ID does not exist" containerID="6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d"
Jul 29 17:11:03.124796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:03.124760    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d"} err="failed to get container status \"6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d\": rpc error: code = NotFound desc = could not find container \"6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d\": container with ID starting with 6862667b31ec94da639cec522915846b98a911da6f03f1b09914345f0a80680d not found: ID does not exist"
Jul 29 17:11:04.703590 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:11:04.703550    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2d584d-b180-4617-9cb4-ae917484f112" path="/var/lib/kubelet/pods/de2d584d-b180-4617-9cb4-ae917484f112/volumes"
Jul 29 17:12:04.919485 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:04.919443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:06.400960 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:06.400917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:07.464300 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:07.464264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:07.482865 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:07.482838    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"]
Jul 29 17:12:07.483174 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:07.483145    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="main" containerID="cri-o://fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7" gracePeriod=30
Jul 29 17:12:07.483336 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:07.483174    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="tokenizer" containerID="cri-o://62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd" gracePeriod=30
Jul 29 17:12:07.550010 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:07.549981    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:08.024511 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.024480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:08.342379 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.342296    2559 generic.go:356] "Generic (PLEG): container finished" podID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerID="fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7" exitCode=0
Jul 29 17:12:08.342379 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.342356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerDied","Data":"fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7"}
Jul 29 17:12:08.732418 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.732390    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:12:08.850665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-cache\") pod \"4aed0aac-a5a9-4348-b812-766c16e6d40a\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") "
Jul 29 17:12:08.850665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-tmp\") pod \"4aed0aac-a5a9-4348-b812-766c16e6d40a\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") "
Jul 29 17:12:08.850665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850645    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-kserve-provision-location\") pod \"4aed0aac-a5a9-4348-b812-766c16e6d40a\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") "
Jul 29 17:12:08.850665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-uds\") pod \"4aed0aac-a5a9-4348-b812-766c16e6d40a\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") "
Jul 29 17:12:08.851006 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wdz2\" (UniqueName: \"kubernetes.io/projected/4aed0aac-a5a9-4348-b812-766c16e6d40a-kube-api-access-2wdz2\") pod \"4aed0aac-a5a9-4348-b812-766c16e6d40a\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") "
Jul 29 17:12:08.851006 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850733    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed0aac-a5a9-4348-b812-766c16e6d40a-tls-certs\") pod \"4aed0aac-a5a9-4348-b812-766c16e6d40a\" (UID: \"4aed0aac-a5a9-4348-b812-766c16e6d40a\") "
Jul 29 17:12:08.851006 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "4aed0aac-a5a9-4348-b812-766c16e6d40a" (UID: "4aed0aac-a5a9-4348-b812-766c16e6d40a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:08.851006 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.850936    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4aed0aac-a5a9-4348-b812-766c16e6d40a" (UID: "4aed0aac-a5a9-4348-b812-766c16e6d40a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:08.851196 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.851021    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "4aed0aac-a5a9-4348-b812-766c16e6d40a" (UID: "4aed0aac-a5a9-4348-b812-766c16e6d40a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:08.851359 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.851332    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4aed0aac-a5a9-4348-b812-766c16e6d40a" (UID: "4aed0aac-a5a9-4348-b812-766c16e6d40a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:12:08.852752 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.852728    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed0aac-a5a9-4348-b812-766c16e6d40a-kube-api-access-2wdz2" (OuterVolumeSpecName: "kube-api-access-2wdz2") pod "4aed0aac-a5a9-4348-b812-766c16e6d40a" (UID: "4aed0aac-a5a9-4348-b812-766c16e6d40a"). InnerVolumeSpecName "kube-api-access-2wdz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:12:08.852866 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.852842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aed0aac-a5a9-4348-b812-766c16e6d40a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aed0aac-a5a9-4348-b812-766c16e6d40a" (UID: "4aed0aac-a5a9-4348-b812-766c16e6d40a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:12:08.951292 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.951260    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:12:08.951292 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.951287    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:12:08.951292 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.951296    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:12:08.951292 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.951305    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4aed0aac-a5a9-4348-b812-766c16e6d40a-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:12:08.951559 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.951313    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdz2\" (UniqueName: \"kubernetes.io/projected/4aed0aac-a5a9-4348-b812-766c16e6d40a-kube-api-access-2wdz2\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:12:08.951559 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:08.951323    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed0aac-a5a9-4348-b812-766c16e6d40a-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:12:09.348355 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.348324    2559 generic.go:356] "Generic (PLEG): container finished" podID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerID="62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd" exitCode=0
Jul 29 17:12:09.348534 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.348380    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"
Jul 29 17:12:09.348534 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.348385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerDied","Data":"62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd"}
Jul 29 17:12:09.348534 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.348429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml" event={"ID":"4aed0aac-a5a9-4348-b812-766c16e6d40a","Type":"ContainerDied","Data":"ba28d5aaa00b5b227298008428f0607711ea5499950b37774782784d419718d4"}
Jul 29 17:12:09.348534 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.348452    2559 scope.go:117] "RemoveContainer" containerID="62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd"
Jul 29 17:12:09.357301 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.357282    2559 scope.go:117] "RemoveContainer" containerID="fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7"
Jul 29 17:12:09.364222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.364203    2559 scope.go:117] "RemoveContainer" containerID="1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984"
Jul 29 17:12:09.370565 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.370544    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"]
Jul 29 17:12:09.371814 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.371800    2559 scope.go:117] "RemoveContainer" containerID="62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd"
Jul 29 17:12:09.372046 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:12:09.372029    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd\": container with ID starting with 62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd not found: ID does not exist" containerID="62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd"
Jul 29 17:12:09.372107 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.372054    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd"} err="failed to get container status \"62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd\": rpc error: code = NotFound desc = could not find container \"62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd\": container with ID starting with 62ea1905de5f5748ac884f2378e00be71322547ca188d2c4f879ae6aca9773dd not found: ID does not exist"
Jul 29 17:12:09.372107 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.372075    2559 scope.go:117] "RemoveContainer" containerID="fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7"
Jul 29 17:12:09.372329 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:12:09.372312    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7\": container with ID starting with fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7 not found: ID does not exist" containerID="fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7"
Jul 29 17:12:09.372365 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.372336    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7"} err="failed to get container status \"fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7\": rpc error: code = NotFound desc = could not find container \"fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7\": container with ID starting with fb483117840a36487b36da7fbbbff3397d2fa2095697d928ac7beaee2279a6b7 not found: ID does not exist"
Jul 29 17:12:09.372365 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.372353    2559 scope.go:117] "RemoveContainer" containerID="1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984"
Jul 29 17:12:09.372587 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:12:09.372572    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984\": container with ID starting with 1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984 not found: ID does not exist" containerID="1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984"
Jul 29 17:12:09.372663 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.372593    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984"} err="failed to get container status \"1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984\": rpc error: code = NotFound desc = could not find container \"1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984\": container with ID starting with 1c5e2095e7f106095a2d9d1336e2b62c51b7006583ae636d860d4c093762d984 not found: ID does not exist"
Jul 29 17:12:09.375567 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:09.375549    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7477555ctkhml"]
Jul 29 17:12:10.703829 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:10.703790    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" path="/var/lib/kubelet/pods/4aed0aac-a5a9-4348-b812-766c16e6d40a/volumes"
Jul 29 17:12:24.467282 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.467245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:24.951800 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.951769    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"]
Jul 29 17:12:24.952029 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952017    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="tokenizer"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952030    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="tokenizer"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952042    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="main"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952048    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="main"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952055    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="main"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952059    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="main"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952066    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="tokenizer"
Jul 29 17:12:24.952073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952071    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="tokenizer"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952092    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="storage-initializer"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952097    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="storage-initializer"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952104    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="storage-initializer"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952110    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="storage-initializer"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952162    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="main"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952171    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2d584d-b180-4617-9cb4-ae917484f112" containerName="tokenizer"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952177    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="main"
Jul 29 17:12:24.952322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.952181    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aed0aac-a5a9-4348-b812-766c16e6d40a" containerName="tokenizer"
Jul 29 17:12:24.956176 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.956157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.959889 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.959866    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 29 17:12:24.959889 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.959866    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-rxw5j\""
Jul 29 17:12:24.960037 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.959890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:12:24.966038 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd114b7-1655-402a-9935-1114a50ad44e-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.966160 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.966160 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.966281 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.966281 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.966281 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hnf\" (UniqueName: \"kubernetes.io/projected/fcd114b7-1655-402a-9935-1114a50ad44e-kube-api-access-57hnf\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:24.966432 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:24.966345    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"]
Jul 29 17:12:25.058608 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.058579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:25.067524 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd114b7-1655-402a-9935-1114a50ad44e-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.067602 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.067602 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067561    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.067671 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.067671 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.067749 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hnf\" (UniqueName: \"kubernetes.io/projected/fcd114b7-1655-402a-9935-1114a50ad44e-kube-api-access-57hnf\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.067923 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.068007 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.067952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.068068 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.068000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.068068 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.068040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.070065 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.070045    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd114b7-1655-402a-9935-1114a50ad44e-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.075502 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.075462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hnf\" (UniqueName: \"kubernetes.io/projected/fcd114b7-1655-402a-9935-1114a50ad44e-kube-api-access-57hnf\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.265699 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.265613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:25.392326 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.392304    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"]
Jul 29 17:12:25.394467 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:12:25.394441    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd114b7_1655_402a_9935_1114a50ad44e.slice/crio-94a5c88f0285bf1dd192697ac47f9ad559716cc44bfc1c9cb5e43eddd973f0af WatchSource:0}: Error finding container 94a5c88f0285bf1dd192697ac47f9ad559716cc44bfc1c9cb5e43eddd973f0af: Status 404 returned error can't find the container with id 94a5c88f0285bf1dd192697ac47f9ad559716cc44bfc1c9cb5e43eddd973f0af
Jul 29 17:12:25.422256 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:25.422229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerStarted","Data":"94a5c88f0285bf1dd192697ac47f9ad559716cc44bfc1c9cb5e43eddd973f0af"}
Jul 29 17:12:26.427158 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:26.427134    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcd114b7-1655-402a-9935-1114a50ad44e" containerID="7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee" exitCode=0
Jul 29 17:12:26.427553 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:26.427179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerDied","Data":"7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee"}
Jul 29 17:12:27.433059 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:27.433027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerStarted","Data":"4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae"}
Jul 29 17:12:27.433462 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:27.433099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerStarted","Data":"c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d"}
Jul 29 17:12:27.458913 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:27.458857    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" podStartSLOduration=2.824694874 podStartE2EDuration="3.458820825s" totalImagesPullingTime="0s" totalInitContainerRuntime="634.125951ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:12:27.457344053 +0000 UTC m=+1265.307377752" watchObservedRunningTime="2026-07-29 17:12:27.458820825 +0000 UTC m=+1265.308854526"
Jul 29 17:12:28.437226 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:28.437191    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:31.876662 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:31.876625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:32.246399 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.246363    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"]
Jul 29 17:12:32.249577 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.249555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.252321 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.252298    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 29 17:12:32.252444 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.252323    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-888fx\""
Jul 29 17:12:32.263425 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.263401    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"]
Jul 29 17:12:32.327574 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.327550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.327705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.327589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddb1816-b322-4a1d-9766-e086800fedfa-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.327705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.327609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmls5\" (UniqueName: \"kubernetes.io/projected/8ddb1816-b322-4a1d-9766-e086800fedfa-kube-api-access-mmls5\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.327705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.327637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.327705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.327682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.327897 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.327708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428225 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428366 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddb1816-b322-4a1d-9766-e086800fedfa-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428366 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmls5\" (UniqueName: \"kubernetes.io/projected/8ddb1816-b322-4a1d-9766-e086800fedfa-kube-api-access-mmls5\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428366 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428366 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428522 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428673 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.428916 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.428836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.430723 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.430701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddb1816-b322-4a1d-9766-e086800fedfa-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.437484 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.437458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmls5\" (UniqueName: \"kubernetes.io/projected/8ddb1816-b322-4a1d-9766-e086800fedfa-kube-api-access-mmls5\") pod \"router-with-refs-test-kserve-router-scheduler-85d876656-57bx2\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.558192 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.558107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:32.640709 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.640598    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:12:32.691060 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:32.691037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"]
Jul 29 17:12:32.693608 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:12:32.693580    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddb1816_b322_4a1d_9766_e086800fedfa.slice/crio-b0c931ac816809287d6680dc5d5f23cbacd1aaf7fa9f9a712b984d5b5e3a6b47 WatchSource:0}: Error finding container b0c931ac816809287d6680dc5d5f23cbacd1aaf7fa9f9a712b984d5b5e3a6b47: Status 404 returned error can't find the container with id b0c931ac816809287d6680dc5d5f23cbacd1aaf7fa9f9a712b984d5b5e3a6b47
Jul 29 17:12:33.457648 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:33.457611    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerID="8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38" exitCode=0
Jul 29 17:12:33.458127 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:33.457670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerDied","Data":"8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38"}
Jul 29 17:12:33.458127 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:33.457717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerStarted","Data":"b0c931ac816809287d6680dc5d5f23cbacd1aaf7fa9f9a712b984d5b5e3a6b47"}
Jul 29 17:12:34.463472 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:34.463437    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerStarted","Data":"f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6"}
Jul 29 17:12:34.463960 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:34.463487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerStarted","Data":"5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7"}
Jul 29 17:12:34.463960 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:34.463517    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:34.486707 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:34.486654    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" podStartSLOduration=1.819443725 podStartE2EDuration="2.486638964s" totalImagesPullingTime="0s" totalInitContainerRuntime="667.195239ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:12:34.485353609 +0000 UTC m=+1272.335387311" watchObservedRunningTime="2026-07-29 17:12:34.486638964 +0000 UTC m=+1272.336672662"
Jul 29 17:12:35.265877 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:35.265842    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:35.266073 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:35.265919    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:35.268697 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:35.268670    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:35.468679 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:35.468643    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:12:42.558279 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:42.558243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:42.558768 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:42.558345    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:42.560843 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:42.560817    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:43.500574 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:43.500540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:12:56.473390 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:12:56.473359    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:13:04.504463 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:13:04.504431    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:13:20.092708 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:13:20.092648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:16:53.818794 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:16:53.818715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:18:15.417894 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:18:15.417812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:19:44.171836 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:19:44.171798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:24:16.168455 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:24:16.168371    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:26:11.414951 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:26:11.414919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:26:22.114520 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:26:22.114483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:26:22.215035 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:26:22.215007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:09.464920 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.464884    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"]
Jul 29 17:27:09.465531 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.465188    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="main" containerID="cri-o://c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d" gracePeriod=30
Jul 29 17:27:09.465531 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.465298    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="tokenizer" containerID="cri-o://4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae" gracePeriod=30
Jul 29 17:27:09.472487 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.472457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:09.535884 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.535851    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:09.699185 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.699150    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcd114b7-1655-402a-9935-1114a50ad44e" containerID="c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d" exitCode=0
Jul 29 17:27:09.699361 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:09.699216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerDied","Data":"c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d"}
Jul 29 17:27:10.605392 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.605370    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:27:10.687287 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687262    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-uds\") pod \"fcd114b7-1655-402a-9935-1114a50ad44e\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") "
Jul 29 17:27:10.687404 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687315    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd114b7-1655-402a-9935-1114a50ad44e-tls-certs\") pod \"fcd114b7-1655-402a-9935-1114a50ad44e\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") "
Jul 29 17:27:10.687404 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-kserve-provision-location\") pod \"fcd114b7-1655-402a-9935-1114a50ad44e\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") "
Jul 29 17:27:10.687481 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687444    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57hnf\" (UniqueName: \"kubernetes.io/projected/fcd114b7-1655-402a-9935-1114a50ad44e-kube-api-access-57hnf\") pod \"fcd114b7-1655-402a-9935-1114a50ad44e\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") "
Jul 29 17:27:10.687481 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-cache\") pod \"fcd114b7-1655-402a-9935-1114a50ad44e\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") "
Jul 29 17:27:10.687580 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687504    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-tmp\") pod \"fcd114b7-1655-402a-9935-1114a50ad44e\" (UID: \"fcd114b7-1655-402a-9935-1114a50ad44e\") "
Jul 29 17:27:10.687580 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687528    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fcd114b7-1655-402a-9935-1114a50ad44e" (UID: "fcd114b7-1655-402a-9935-1114a50ad44e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:10.687721 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687692    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:10.687796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "fcd114b7-1655-402a-9935-1114a50ad44e" (UID: "fcd114b7-1655-402a-9935-1114a50ad44e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:10.687860 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.687844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "fcd114b7-1655-402a-9935-1114a50ad44e" (UID: "fcd114b7-1655-402a-9935-1114a50ad44e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:10.688173 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.688152    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcd114b7-1655-402a-9935-1114a50ad44e" (UID: "fcd114b7-1655-402a-9935-1114a50ad44e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:10.689431 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.689410    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd114b7-1655-402a-9935-1114a50ad44e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcd114b7-1655-402a-9935-1114a50ad44e" (UID: "fcd114b7-1655-402a-9935-1114a50ad44e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:27:10.689522 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.689506    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd114b7-1655-402a-9935-1114a50ad44e-kube-api-access-57hnf" (OuterVolumeSpecName: "kube-api-access-57hnf") pod "fcd114b7-1655-402a-9935-1114a50ad44e" (UID: "fcd114b7-1655-402a-9935-1114a50ad44e"). InnerVolumeSpecName "kube-api-access-57hnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:27:10.704843 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.704821    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcd114b7-1655-402a-9935-1114a50ad44e" containerID="4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae" exitCode=0
Jul 29 17:27:10.704936 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.704863    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"
Jul 29 17:27:10.704936 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.704863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerDied","Data":"4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae"}
Jul 29 17:27:10.705024 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.704980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg" event={"ID":"fcd114b7-1655-402a-9935-1114a50ad44e","Type":"ContainerDied","Data":"94a5c88f0285bf1dd192697ac47f9ad559716cc44bfc1c9cb5e43eddd973f0af"}
Jul 29 17:27:10.705024 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.705010    2559 scope.go:117] "RemoveContainer" containerID="4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae"
Jul 29 17:27:10.713592 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.713571    2559 scope.go:117] "RemoveContainer" containerID="c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d"
Jul 29 17:27:10.721067 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.721052    2559 scope.go:117] "RemoveContainer" containerID="7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee"
Jul 29 17:27:10.727953 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.727935    2559 scope.go:117] "RemoveContainer" containerID="4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae"
Jul 29 17:27:10.728069 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.728049    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"]
Jul 29 17:27:10.728200 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:27:10.728183    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae\": container with ID starting with 4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae not found: ID does not exist" containerID="4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae"
Jul 29 17:27:10.728372 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.728206    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae"} err="failed to get container status \"4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae\": rpc error: code = NotFound desc = could not find container \"4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae\": container with ID starting with 4a6ffcb180583107e618df41facb237bf6000fa8ef33eee16588a1e8c900b3ae not found: ID does not exist"
Jul 29 17:27:10.728372 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.728222    2559 scope.go:117] "RemoveContainer" containerID="c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d"
Jul 29 17:27:10.728461 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:27:10.728445    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d\": container with ID starting with c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d not found: ID does not exist" containerID="c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d"
Jul 29 17:27:10.728501 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.728464    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d"} err="failed to get container status \"c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d\": rpc error: code = NotFound desc = could not find container \"c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d\": container with ID starting with c5539e1dcc06a718838ac3a99ae16b56f9f53799b22cf6a341f618995c80ef8d not found: ID does not exist"
Jul 29 17:27:10.728501 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.728480    2559 scope.go:117] "RemoveContainer" containerID="7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee"
Jul 29 17:27:10.728721 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:27:10.728706    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee\": container with ID starting with 7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee not found: ID does not exist" containerID="7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee"
Jul 29 17:27:10.728773 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.728725    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee"} err="failed to get container status \"7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee\": rpc error: code = NotFound desc = could not find container \"7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee\": container with ID starting with 7eea9583cfeaa1eb14fa6d2ccc044f4843277b78c5b50482fad433c592c17bee not found: ID does not exist"
Jul 29 17:27:10.731804 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.731784    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75576487bhnxvg"]
Jul 29 17:27:10.788485 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.788465    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:10.788485 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.788483    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57hnf\" (UniqueName: \"kubernetes.io/projected/fcd114b7-1655-402a-9935-1114a50ad44e-kube-api-access-57hnf\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:10.788647 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.788493    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:10.788647 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.788505    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcd114b7-1655-402a-9935-1114a50ad44e-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:10.788647 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:10.788518    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd114b7-1655-402a-9935-1114a50ad44e-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:12.704797 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:12.704764    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" path="/var/lib/kubelet/pods/fcd114b7-1655-402a-9935-1114a50ad44e/volumes"
Jul 29 17:27:16.881360 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:16.881286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:18.782001 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:18.781957    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"]
Jul 29 17:27:18.782483 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:18.782267    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="main" containerID="cri-o://5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7" gracePeriod=30
Jul 29 17:27:18.782483 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:18.782322    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="tokenizer" containerID="cri-o://f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6" gracePeriod=30
Jul 29 17:27:18.798595 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:18.798554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:18.898284 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:18.898258    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:19.739724 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:19.739691    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerID="5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7" exitCode=0
Jul 29 17:27:19.739915 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:19.739767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerDied","Data":"5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7"}
Jul 29 17:27:20.043619 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.043593    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:27:20.175379 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddb1816-b322-4a1d-9766-e086800fedfa-tls-certs\") pod \"8ddb1816-b322-4a1d-9766-e086800fedfa\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") "
Jul 29 17:27:20.175379 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-tmp\") pod \"8ddb1816-b322-4a1d-9766-e086800fedfa\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") "
Jul 29 17:27:20.175625 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175452    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-uds\") pod \"8ddb1816-b322-4a1d-9766-e086800fedfa\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") "
Jul 29 17:27:20.175625 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175469    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmls5\" (UniqueName: \"kubernetes.io/projected/8ddb1816-b322-4a1d-9766-e086800fedfa-kube-api-access-mmls5\") pod \"8ddb1816-b322-4a1d-9766-e086800fedfa\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") "
Jul 29 17:27:20.175625 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175501    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-kserve-provision-location\") pod \"8ddb1816-b322-4a1d-9766-e086800fedfa\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") "
Jul 29 17:27:20.175625 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-cache\") pod \"8ddb1816-b322-4a1d-9766-e086800fedfa\" (UID: \"8ddb1816-b322-4a1d-9766-e086800fedfa\") "
Jul 29 17:27:20.175815 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175701    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8ddb1816-b322-4a1d-9766-e086800fedfa" (UID: "8ddb1816-b322-4a1d-9766-e086800fedfa"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:20.175815 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175752    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "8ddb1816-b322-4a1d-9766-e086800fedfa" (UID: "8ddb1816-b322-4a1d-9766-e086800fedfa"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:20.175815 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.175801    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8ddb1816-b322-4a1d-9766-e086800fedfa" (UID: "8ddb1816-b322-4a1d-9766-e086800fedfa"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:20.176180 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.176157    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ddb1816-b322-4a1d-9766-e086800fedfa" (UID: "8ddb1816-b322-4a1d-9766-e086800fedfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:27:20.177552 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.177530    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddb1816-b322-4a1d-9766-e086800fedfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ddb1816-b322-4a1d-9766-e086800fedfa" (UID: "8ddb1816-b322-4a1d-9766-e086800fedfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:27:20.177612 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.177583    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddb1816-b322-4a1d-9766-e086800fedfa-kube-api-access-mmls5" (OuterVolumeSpecName: "kube-api-access-mmls5") pod "8ddb1816-b322-4a1d-9766-e086800fedfa" (UID: "8ddb1816-b322-4a1d-9766-e086800fedfa"). InnerVolumeSpecName "kube-api-access-mmls5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:27:20.276690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.276611    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddb1816-b322-4a1d-9766-e086800fedfa-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:20.276690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.276637    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:20.276690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.276646    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:20.276690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.276661    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmls5\" (UniqueName: \"kubernetes.io/projected/8ddb1816-b322-4a1d-9766-e086800fedfa-kube-api-access-mmls5\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:20.276690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.276673    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:20.276690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.276681    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddb1816-b322-4a1d-9766-e086800fedfa-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:27:20.744993 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.744967    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerID="f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6" exitCode=0
Jul 29 17:27:20.745178 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.745020    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"
Jul 29 17:27:20.745178 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.745040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerDied","Data":"f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6"}
Jul 29 17:27:20.745178 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.745104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2" event={"ID":"8ddb1816-b322-4a1d-9766-e086800fedfa","Type":"ContainerDied","Data":"b0c931ac816809287d6680dc5d5f23cbacd1aaf7fa9f9a712b984d5b5e3a6b47"}
Jul 29 17:27:20.745178 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.745137    2559 scope.go:117] "RemoveContainer" containerID="f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6"
Jul 29 17:27:20.764069 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.764047    2559 scope.go:117] "RemoveContainer" containerID="5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7"
Jul 29 17:27:20.767005 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.766974    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"]
Jul 29 17:27:20.771451 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.771427    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85d876656-57bx2"]
Jul 29 17:27:20.777239 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.777208    2559 scope.go:117] "RemoveContainer" containerID="8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38"
Jul 29 17:27:20.784141 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.784123    2559 scope.go:117] "RemoveContainer" containerID="f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6"
Jul 29 17:27:20.784385 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:27:20.784366    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6\": container with ID starting with f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6 not found: ID does not exist" containerID="f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6"
Jul 29 17:27:20.784446 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.784392    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6"} err="failed to get container status \"f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6\": rpc error: code = NotFound desc = could not find container \"f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6\": container with ID starting with f1317c0ca91b5c85a30ad1d9c48bb8141d93f077d1abeab38fc22836f1ef94b6 not found: ID does not exist"
Jul 29 17:27:20.784446 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.784410    2559 scope.go:117] "RemoveContainer" containerID="5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7"
Jul 29 17:27:20.784611 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:27:20.784595    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7\": container with ID starting with 5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7 not found: ID does not exist" containerID="5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7"
Jul 29 17:27:20.784646 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.784616    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7"} err="failed to get container status \"5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7\": rpc error: code = NotFound desc = could not find container \"5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7\": container with ID starting with 5912adbdfa5714516cc5b113b06141ba2aeea0319dacc2f679cdc574f49a0ab7 not found: ID does not exist"
Jul 29 17:27:20.784646 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.784631    2559 scope.go:117] "RemoveContainer" containerID="8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38"
Jul 29 17:27:20.784877 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:27:20.784857    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38\": container with ID starting with 8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38 not found: ID does not exist" containerID="8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38"
Jul 29 17:27:20.784914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:20.784884    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38"} err="failed to get container status \"8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38\": rpc error: code = NotFound desc = could not find container \"8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38\": container with ID starting with 8d046bd6740572b2ccf64d040cb4b8d045579b4ee41bbd42d10da6168a16dc38 not found: ID does not exist"
Jul 29 17:27:22.704483 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:22.704454    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" path="/var/lib/kubelet/pods/8ddb1816-b322-4a1d-9766-e086800fedfa/volumes"
Jul 29 17:27:26.146795 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.146761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:26.331911 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.331875    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"]
Jul 29 17:27:26.332201 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332188    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="main"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332202    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="main"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332214    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="storage-initializer"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332220    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="storage-initializer"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332231    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="tokenizer"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332237    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="tokenizer"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332244    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="main"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332248    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="main"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332258    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="storage-initializer"
Jul 29 17:27:26.332261 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332263    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="storage-initializer"
Jul 29 17:27:26.332557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332271    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="tokenizer"
Jul 29 17:27:26.332557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332276    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="tokenizer"
Jul 29 17:27:26.332557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332315    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="tokenizer"
Jul 29 17:27:26.332557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332321    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="tokenizer"
Jul 29 17:27:26.332557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332327    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddb1816-b322-4a1d-9766-e086800fedfa" containerName="main"
Jul 29 17:27:26.332557 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.332335    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd114b7-1655-402a-9935-1114a50ad44e" containerName="main"
Jul 29 17:27:26.336836 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.336819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.339749 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.339721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jul 29 17:27:26.339867 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.339769    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-szzmj\""
Jul 29 17:27:26.339867 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.339855    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-lw48q\""
Jul 29 17:27:26.348416 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.348397    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"]
Jul 29 17:27:26.379848 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.379824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:26.528373 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.528348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.528603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.528383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7t7\" (UniqueName: \"kubernetes.io/projected/8a70445e-6320-4088-94f8-6fdb9acad359-kube-api-access-2j7t7\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.528603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.528474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.528603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.528503    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a70445e-6320-4088-94f8-6fdb9acad359-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.528603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.528527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.528603 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.528568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.629657 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.629757 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7t7\" (UniqueName: \"kubernetes.io/projected/8a70445e-6320-4088-94f8-6fdb9acad359-kube-api-access-2j7t7\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.629809 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.629846 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a70445e-6320-4088-94f8-6fdb9acad359-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.629899 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.629899 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.630018 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.629992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.630151 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.630132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.630224 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.630208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.630271 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.630243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.632196 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.632165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a70445e-6320-4088-94f8-6fdb9acad359-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.640030 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.640003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7t7\" (UniqueName: \"kubernetes.io/projected/8a70445e-6320-4088-94f8-6fdb9acad359-kube-api-access-2j7t7\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.645786 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.645766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:26.808722 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:26.808696    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"]
Jul 29 17:27:26.811375 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:27:26.811341    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a70445e_6320_4088_94f8_6fdb9acad359.slice/crio-a26ca2bf91cb78981ec289f4d3c82a95167a9088ed63b896c96567e3753c929c WatchSource:0}: Error finding container a26ca2bf91cb78981ec289f4d3c82a95167a9088ed63b896c96567e3753c929c: Status 404 returned error can't find the container with id a26ca2bf91cb78981ec289f4d3c82a95167a9088ed63b896c96567e3753c929c
Jul 29 17:27:27.774963 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:27.774928    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a70445e-6320-4088-94f8-6fdb9acad359" containerID="d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a" exitCode=0
Jul 29 17:27:27.775391 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:27.775020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerDied","Data":"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"}
Jul 29 17:27:27.775391 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:27.775074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerStarted","Data":"a26ca2bf91cb78981ec289f4d3c82a95167a9088ed63b896c96567e3753c929c"}
Jul 29 17:27:28.781407 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:28.781374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerStarted","Data":"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"}
Jul 29 17:27:28.781821 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:28.781423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerStarted","Data":"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"}
Jul 29 17:27:28.781821 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:28.781502    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:28.804222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:28.804166    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" podStartSLOduration=2.120864529 podStartE2EDuration="2.804150702s" totalImagesPullingTime="0s" totalInitContainerRuntime="683.286173ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:27:28.802438546 +0000 UTC m=+2166.652472235" watchObservedRunningTime="2026-07-29 17:27:28.804150702 +0000 UTC m=+2166.654184400"
Jul 29 17:27:36.646688 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:36.646645    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:36.647140 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:36.646720    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:36.649463 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:36.649438    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:36.811151 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:36.811124    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:27:51.509100 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:51.509054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:27:57.815511 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:27:57.815482    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:29:44.376868 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:29:44.376833    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:30:38.850033 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:38.849995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:30:54.389553 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.389522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:30:54.557193 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.557155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:30:54.715172 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.715133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:30:54.808677 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.808648    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jul 29 17:30:54.811833 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.811817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.814302 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.814282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-tj6gb\""
Jul 29 17:30:54.815419 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.815383    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jul 29 17:30:54.833335 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.833306    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jul 29 17:30:54.834792 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.834766    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"]
Jul 29 17:30:54.838532 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.838514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.841370 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.841347    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-5fjln\""
Jul 29 17:30:54.848979 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.848959    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"]
Jul 29 17:30:54.874491 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.874463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:30:54.894263 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4183c4be-6f99-4304-aa37-0e16425653a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.894408 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtpz\" (UniqueName: \"kubernetes.io/projected/4183c4be-6f99-4304-aa37-0e16425653a2-kube-api-access-lvtpz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.894408 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894296    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.894408 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.894408 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.894408 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894405    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.894598 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.894433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.995865 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.995865 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.995865 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.995865 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.995974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996076    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9f8\" (UniqueName: \"kubernetes.io/projected/c5114605-a915-4df0-906c-642dbdc35f35-kube-api-access-jd9f8\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996120    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996233 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5114605-a915-4df0-906c-642dbdc35f35-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:54.996544 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4183c4be-6f99-4304-aa37-0e16425653a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996544 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996544 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtpz\" (UniqueName: \"kubernetes.io/projected/4183c4be-6f99-4304-aa37-0e16425653a2-kube-api-access-lvtpz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.996544 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.996315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.998097 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.998065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:54.998480 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:54.998464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4183c4be-6f99-4304-aa37-0e16425653a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:55.005344 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.005322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtpz\" (UniqueName: \"kubernetes.io/projected/4183c4be-6f99-4304-aa37-0e16425653a2-kube-api-access-lvtpz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:55.097399 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097399 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097623 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097623 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9f8\" (UniqueName: \"kubernetes.io/projected/c5114605-a915-4df0-906c-642dbdc35f35-kube-api-access-jd9f8\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097623 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097623 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5114605-a915-4df0-906c-642dbdc35f35-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097830 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097889 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097936 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.097976 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.097936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.099865 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.099849    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5114605-a915-4df0-906c-642dbdc35f35-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.106340 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.106316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9f8\" (UniqueName: \"kubernetes.io/projected/c5114605-a915-4df0-906c-642dbdc35f35-kube-api-access-jd9f8\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.122012 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.121988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:30:55.148222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.148199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:55.266907 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.266883    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jul 29 17:30:55.269549 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:30:55.269515    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4183c4be_6f99_4304_aa37_0e16425653a2.slice/crio-191571ac35e0a475722410b858f209bf8c0dcab8f18a458ebed28401d1df489e WatchSource:0}: Error finding container 191571ac35e0a475722410b858f209bf8c0dcab8f18a458ebed28401d1df489e: Status 404 returned error can't find the container with id 191571ac35e0a475722410b858f209bf8c0dcab8f18a458ebed28401d1df489e
Jul 29 17:30:55.298560 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.298534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"]
Jul 29 17:30:55.302536 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:30:55.302503    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5114605_a915_4df0_906c_642dbdc35f35.slice/crio-3b02ab40b4ba5fc68c48336fdc3e1a68b96d03c711e25807741206f7127eb052 WatchSource:0}: Error finding container 3b02ab40b4ba5fc68c48336fdc3e1a68b96d03c711e25807741206f7127eb052: Status 404 returned error can't find the container with id 3b02ab40b4ba5fc68c48336fdc3e1a68b96d03c711e25807741206f7127eb052
Jul 29 17:30:55.535130 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.535019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerStarted","Data":"756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75"}
Jul 29 17:30:55.535130 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.535107    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerStarted","Data":"3b02ab40b4ba5fc68c48336fdc3e1a68b96d03c711e25807741206f7127eb052"}
Jul 29 17:30:55.537162 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.537125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"4183c4be-6f99-4304-aa37-0e16425653a2","Type":"ContainerStarted","Data":"5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce"}
Jul 29 17:30:55.537319 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:55.537185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"4183c4be-6f99-4304-aa37-0e16425653a2","Type":"ContainerStarted","Data":"191571ac35e0a475722410b858f209bf8c0dcab8f18a458ebed28401d1df489e"}
Jul 29 17:30:56.543746 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:56.543705    2559 generic.go:356] "Generic (PLEG): container finished" podID="c5114605-a915-4df0-906c-642dbdc35f35" containerID="756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75" exitCode=0
Jul 29 17:30:56.544197 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:56.543792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerDied","Data":"756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75"}
Jul 29 17:30:57.550373 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:57.550329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerStarted","Data":"44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb"}
Jul 29 17:30:57.550827 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:57.550392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerStarted","Data":"483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48"}
Jul 29 17:30:57.550827 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:57.550425    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:30:57.575054 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:30:57.575004    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" podStartSLOduration=2.865326767 podStartE2EDuration="3.574989227s" totalImagesPullingTime="0s" totalInitContainerRuntime="709.66246ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:30:57.572683536 +0000 UTC m=+2375.422717234" watchObservedRunningTime="2026-07-29 17:30:57.574989227 +0000 UTC m=+2375.425022925"
Jul 29 17:31:01.570914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:01.570881    2559 generic.go:356] "Generic (PLEG): container finished" podID="4183c4be-6f99-4304-aa37-0e16425653a2" containerID="5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce" exitCode=0
Jul 29 17:31:01.571434 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:01.570964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"4183c4be-6f99-4304-aa37-0e16425653a2","Type":"ContainerDied","Data":"5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce"}
Jul 29 17:31:01.572170 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:01.572151    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 17:31:05.149162 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:05.149121    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:31:05.149636 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:05.149493    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:31:05.150638 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:05.150607    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="tokenizer" probeResult="failure" output="Get \"http://10.133.0.36:8082/healthz\": dial tcp 10.133.0.36:8082: connect: connection refused"
Jul 29 17:31:09.758375 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:09.758318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:31:15.150419 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:15.150388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:31:15.151747 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:15.151719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:31:29.557660 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.557623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:31:29.844577 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.844494    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"]
Jul 29 17:31:29.852964 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.852938    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:29.855958 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.855937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 29 17:31:29.856136 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.855981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-295t8\""
Jul 29 17:31:29.863425 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.861983    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"]
Jul 29 17:31:29.871252 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.871217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:31:29.902369 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.902340    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:29.902530 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.902410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsxh\" (UniqueName: \"kubernetes.io/projected/21c84606-1cf2-4a34-b13d-452b6d531ff3-kube-api-access-bxsxh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:29.902530 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.902479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c84606-1cf2-4a34-b13d-452b6d531ff3-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:29.902530 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.902519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:29.902669 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.902539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:29.902669 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:29.902596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004011 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.003973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsxh\" (UniqueName: \"kubernetes.io/projected/21c84606-1cf2-4a34-b13d-452b6d531ff3-kube-api-access-bxsxh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c84606-1cf2-4a34-b13d-452b6d531ff3-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004222 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004451 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004451 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004444    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004556 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004733 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.004797 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.004742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.007015 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.006987    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c84606-1cf2-4a34-b13d-452b6d531ff3-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.015916 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.015866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsxh\" (UniqueName: \"kubernetes.io/projected/21c84606-1cf2-4a34-b13d-452b6d531ff3-kube-api-access-bxsxh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:30.167125 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:30.167011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:32.276616 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:32.276580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:31:32.294858 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:32.294829    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"]
Jul 29 17:31:32.295104 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:32.295058    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="main" containerID="cri-o://1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069" gracePeriod=30
Jul 29 17:31:32.295208 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:32.295145    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="tokenizer" containerID="cri-o://0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028" gracePeriod=30
Jul 29 17:31:32.335727 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:32.335682    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:31:35.640632 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:35.640601    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:31:36.811371 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:36.811328    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="tokenizer" probeResult="failure" output="Get \"http://10.133.0.34:8082/healthz\": dial tcp 10.133.0.34:8082: connect: connection refused"
Jul 29 17:31:37.814034 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:31:37.814002    2559 logging.go:55] [core] [Channel #1405 SubChannel #1406]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.34:9003", ServerName: "10.133.0.34:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.34:9003: connect: connection refused"
Jul 29 17:31:38.814451 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:38.814409    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.34:9003\" within 1s: context deadline exceeded"
Jul 29 17:31:38.814877 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:31:38.814516    2559 logging.go:55] [core] [Channel #1405 SubChannel #1406]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.34:9003", ServerName: "10.133.0.34:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.34:9003: connect: connection refused"
Jul 29 17:31:39.167405 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.167370    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:31:39.289637 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289606    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-uds\") pod \"8a70445e-6320-4088-94f8-6fdb9acad359\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") "
Jul 29 17:31:39.289811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-cache\") pod \"8a70445e-6320-4088-94f8-6fdb9acad359\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") "
Jul 29 17:31:39.289811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289701    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j7t7\" (UniqueName: \"kubernetes.io/projected/8a70445e-6320-4088-94f8-6fdb9acad359-kube-api-access-2j7t7\") pod \"8a70445e-6320-4088-94f8-6fdb9acad359\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") "
Jul 29 17:31:39.289811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289724    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a70445e-6320-4088-94f8-6fdb9acad359-tls-certs\") pod \"8a70445e-6320-4088-94f8-6fdb9acad359\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") "
Jul 29 17:31:39.289811 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289760    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-tmp\") pod \"8a70445e-6320-4088-94f8-6fdb9acad359\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") "
Jul 29 17:31:39.290016 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289830    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-kserve-provision-location\") pod \"8a70445e-6320-4088-94f8-6fdb9acad359\" (UID: \"8a70445e-6320-4088-94f8-6fdb9acad359\") "
Jul 29 17:31:39.290016 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289948    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8a70445e-6320-4088-94f8-6fdb9acad359" (UID: "8a70445e-6320-4088-94f8-6fdb9acad359"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:39.290016 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.289988    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8a70445e-6320-4088-94f8-6fdb9acad359" (UID: "8a70445e-6320-4088-94f8-6fdb9acad359"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:39.290200 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.290164    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:31:39.290200 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.290175    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "8a70445e-6320-4088-94f8-6fdb9acad359" (UID: "8a70445e-6320-4088-94f8-6fdb9acad359"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:39.290294 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.290181    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:31:39.290639 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.290605    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a70445e-6320-4088-94f8-6fdb9acad359" (UID: "8a70445e-6320-4088-94f8-6fdb9acad359"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:31:39.292158 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.292132    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a70445e-6320-4088-94f8-6fdb9acad359-kube-api-access-2j7t7" (OuterVolumeSpecName: "kube-api-access-2j7t7") pod "8a70445e-6320-4088-94f8-6fdb9acad359" (UID: "8a70445e-6320-4088-94f8-6fdb9acad359"). InnerVolumeSpecName "kube-api-access-2j7t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:31:39.292412 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.292388    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a70445e-6320-4088-94f8-6fdb9acad359-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a70445e-6320-4088-94f8-6fdb9acad359" (UID: "8a70445e-6320-4088-94f8-6fdb9acad359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:31:39.390914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.390851    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:31:39.390914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.390888    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7t7\" (UniqueName: \"kubernetes.io/projected/8a70445e-6320-4088-94f8-6fdb9acad359-kube-api-access-2j7t7\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:31:39.390914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.390900    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a70445e-6320-4088-94f8-6fdb9acad359-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:31:39.390914 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.390909    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a70445e-6320-4088-94f8-6fdb9acad359-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:31:39.746145 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746108    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a70445e-6320-4088-94f8-6fdb9acad359" containerID="0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028" exitCode=0
Jul 29 17:31:39.746145 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746138    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a70445e-6320-4088-94f8-6fdb9acad359" containerID="1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069" exitCode=0
Jul 29 17:31:39.746393 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746154    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerDied","Data":"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"}
Jul 29 17:31:39.746393 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerDied","Data":"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"}
Jul 29 17:31:39.746393 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn" event={"ID":"8a70445e-6320-4088-94f8-6fdb9acad359","Type":"ContainerDied","Data":"a26ca2bf91cb78981ec289f4d3c82a95167a9088ed63b896c96567e3753c929c"}
Jul 29 17:31:39.746393 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746169    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"
Jul 29 17:31:39.746393 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.746257    2559 scope.go:117] "RemoveContainer" containerID="0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"
Jul 29 17:31:39.786902 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.786872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"]
Jul 29 17:31:39.790649 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:39.790625    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schemm9kn"]
Jul 29 17:31:40.350456 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.350432    2559 scope.go:117] "RemoveContainer" containerID="1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"
Jul 29 17:31:40.363884 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.363863    2559 scope.go:117] "RemoveContainer" containerID="d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"
Jul 29 17:31:40.371605 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.371568    2559 scope.go:117] "RemoveContainer" containerID="0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"
Jul 29 17:31:40.371912 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:31:40.371884    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028\": container with ID starting with 0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028 not found: ID does not exist" containerID="0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"
Jul 29 17:31:40.372054 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.371920    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"} err="failed to get container status \"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028\": rpc error: code = NotFound desc = could not find container \"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028\": container with ID starting with 0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028 not found: ID does not exist"
Jul 29 17:31:40.372054 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.371941    2559 scope.go:117] "RemoveContainer" containerID="1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"
Jul 29 17:31:40.372238 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:31:40.372217    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069\": container with ID starting with 1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069 not found: ID does not exist" containerID="1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"
Jul 29 17:31:40.372282 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.372245    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"} err="failed to get container status \"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069\": rpc error: code = NotFound desc = could not find container \"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069\": container with ID starting with 1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069 not found: ID does not exist"
Jul 29 17:31:40.372282 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.372259    2559 scope.go:117] "RemoveContainer" containerID="d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"
Jul 29 17:31:40.372520 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:31:40.372495    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a\": container with ID starting with d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a not found: ID does not exist" containerID="d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"
Jul 29 17:31:40.372604 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.372523    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"} err="failed to get container status \"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a\": rpc error: code = NotFound desc = could not find container \"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a\": container with ID starting with d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a not found: ID does not exist"
Jul 29 17:31:40.372604 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.372537    2559 scope.go:117] "RemoveContainer" containerID="0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"
Jul 29 17:31:40.372810 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.372795    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028"} err="failed to get container status \"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028\": rpc error: code = NotFound desc = could not find container \"0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028\": container with ID starting with 0e720d3501b4c6d58634895ba7a6c623bd7845e7a308022d00b905b42f1b1028 not found: ID does not exist"
Jul 29 17:31:40.372866 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.372810    2559 scope.go:117] "RemoveContainer" containerID="1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"
Jul 29 17:31:40.373095 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.373059    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069"} err="failed to get container status \"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069\": rpc error: code = NotFound desc = could not find container \"1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069\": container with ID starting with 1ac4cd9fa75673d8f3ad79819cfc1b5c3f2f12d6495b9d7c0ad79478179cf069 not found: ID does not exist"
Jul 29 17:31:40.373168 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.373153    2559 scope.go:117] "RemoveContainer" containerID="d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"
Jul 29 17:31:40.373409 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.373391    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a"} err="failed to get container status \"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a\": rpc error: code = NotFound desc = could not find container \"d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a\": container with ID starting with d1612c8a2959e96d20107a3cb160f6bee8deae9f6b7a4c84d9696af379d7b10a not found: ID does not exist"
Jul 29 17:31:40.469326 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.469303    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"]
Jul 29 17:31:40.471909 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:31:40.471882    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c84606_1cf2_4a34_b13d_452b6d531ff3.slice/crio-0f60c2b03a1359714eb9200afd8a0af84cea07d2a3e673490b45b7daa8051d85 WatchSource:0}: Error finding container 0f60c2b03a1359714eb9200afd8a0af84cea07d2a3e673490b45b7daa8051d85: Status 404 returned error can't find the container with id 0f60c2b03a1359714eb9200afd8a0af84cea07d2a3e673490b45b7daa8051d85
Jul 29 17:31:40.706645 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.706603    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" path="/var/lib/kubelet/pods/8a70445e-6320-4088-94f8-6fdb9acad359/volumes"
Jul 29 17:31:40.751857 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.751818    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerStarted","Data":"59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26"}
Jul 29 17:31:40.752032 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.751888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerStarted","Data":"0f60c2b03a1359714eb9200afd8a0af84cea07d2a3e673490b45b7daa8051d85"}
Jul 29 17:31:40.754006 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.753982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"4183c4be-6f99-4304-aa37-0e16425653a2","Type":"ContainerStarted","Data":"6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de"}
Jul 29 17:31:40.789676 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:40.789622    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podStartSLOduration=2.348130464 podStartE2EDuration="46.789603022s" totalImagesPullingTime="38.960152238s" totalInitContainerRuntime="5.48132032s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:31:40.788591781 +0000 UTC m=+2418.638625479" watchObservedRunningTime="2026-07-29 17:31:40.789603022 +0000 UTC m=+2418.639636723"
Jul 29 17:31:41.759664 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:41.759627    2559 generic.go:356] "Generic (PLEG): container finished" podID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerID="59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26" exitCode=0
Jul 29 17:31:41.760047 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:41.759707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerDied","Data":"59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26"}
Jul 29 17:31:42.765969 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:42.765935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerStarted","Data":"57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8"}
Jul 29 17:31:42.766492 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:42.765987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerStarted","Data":"0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61"}
Jul 29 17:31:42.766492 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:42.766095    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:42.788774 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:42.788701    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" podStartSLOduration=13.136223035 podStartE2EDuration="13.788678488s" totalImagesPullingTime="0s" totalInitContainerRuntime="652.455453ms" isStatefulPod=false podCreationTimestamp="2026-07-29 17:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:31:42.787344749 +0000 UTC m=+2420.637378449" watchObservedRunningTime="2026-07-29 17:31:42.788678488 +0000 UTC m=+2420.638712187"
Jul 29 17:31:50.167895 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:50.167799    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:50.167895 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:50.167886    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:50.170799 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:50.170774    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:31:50.801267 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:31:50.801206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:32:03.912555 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:32:03.912519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:32:10.961480 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:32:10.961445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:32:11.805735 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:32:11.805708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:33:32.533012 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:32.532975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:33:32.540766 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:32.540739    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"]
Jul 29 17:33:32.541242 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:32.540975    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="main" containerID="cri-o://483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48" gracePeriod=30
Jul 29 17:33:32.541242 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:32.541012    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="tokenizer" containerID="cri-o://44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb" gracePeriod=30
Jul 29 17:33:32.590887 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:32.590854    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:33:33.223423 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.223390    2559 generic.go:356] "Generic (PLEG): container finished" podID="c5114605-a915-4df0-906c-642dbdc35f35" containerID="483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48" exitCode=0
Jul 29 17:33:33.223635 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.223439    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerDied","Data":"483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48"}
Jul 29 17:33:33.698820 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.698792    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:33:33.735658 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735633    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-cache\") pod \"c5114605-a915-4df0-906c-642dbdc35f35\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") "
Jul 29 17:33:33.735796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735674    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-uds\") pod \"c5114605-a915-4df0-906c-642dbdc35f35\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") "
Jul 29 17:33:33.735796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735692    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-tmp\") pod \"c5114605-a915-4df0-906c-642dbdc35f35\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") "
Jul 29 17:33:33.735796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-kserve-provision-location\") pod \"c5114605-a915-4df0-906c-642dbdc35f35\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") "
Jul 29 17:33:33.735796 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5114605-a915-4df0-906c-642dbdc35f35-tls-certs\") pod \"c5114605-a915-4df0-906c-642dbdc35f35\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") "
Jul 29 17:33:33.736015 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd9f8\" (UniqueName: \"kubernetes.io/projected/c5114605-a915-4df0-906c-642dbdc35f35-kube-api-access-jd9f8\") pod \"c5114605-a915-4df0-906c-642dbdc35f35\" (UID: \"c5114605-a915-4df0-906c-642dbdc35f35\") "
Jul 29 17:33:33.736015 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "c5114605-a915-4df0-906c-642dbdc35f35" (UID: "c5114605-a915-4df0-906c-642dbdc35f35"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:33.736015 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.735944    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c5114605-a915-4df0-906c-642dbdc35f35" (UID: "c5114605-a915-4df0-906c-642dbdc35f35"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:33.736205 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.736071    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "c5114605-a915-4df0-906c-642dbdc35f35" (UID: "c5114605-a915-4df0-906c-642dbdc35f35"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:33.736272 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.736215    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:33.736272 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.736234    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:33.736272 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.736246    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:33.736570 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.736549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5114605-a915-4df0-906c-642dbdc35f35" (UID: "c5114605-a915-4df0-906c-642dbdc35f35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:33.737940 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.737921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5114605-a915-4df0-906c-642dbdc35f35-kube-api-access-jd9f8" (OuterVolumeSpecName: "kube-api-access-jd9f8") pod "c5114605-a915-4df0-906c-642dbdc35f35" (UID: "c5114605-a915-4df0-906c-642dbdc35f35"). InnerVolumeSpecName "kube-api-access-jd9f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:33:33.738098 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.738068    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5114605-a915-4df0-906c-642dbdc35f35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5114605-a915-4df0-906c-642dbdc35f35" (UID: "c5114605-a915-4df0-906c-642dbdc35f35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:33:33.837372 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.837307    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5114605-a915-4df0-906c-642dbdc35f35-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:33.837372 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.837332    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5114605-a915-4df0-906c-642dbdc35f35-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:33.837372 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:33.837345    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd9f8\" (UniqueName: \"kubernetes.io/projected/c5114605-a915-4df0-906c-642dbdc35f35-kube-api-access-jd9f8\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:34.229789 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.229753    2559 generic.go:356] "Generic (PLEG): container finished" podID="c5114605-a915-4df0-906c-642dbdc35f35" containerID="44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb" exitCode=0
Jul 29 17:33:34.229982 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.229812    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"
Jul 29 17:33:34.229982 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.229842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerDied","Data":"44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb"}
Jul 29 17:33:34.229982 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.229908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696" event={"ID":"c5114605-a915-4df0-906c-642dbdc35f35","Type":"ContainerDied","Data":"3b02ab40b4ba5fc68c48336fdc3e1a68b96d03c711e25807741206f7127eb052"}
Jul 29 17:33:34.229982 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.229934    2559 scope.go:117] "RemoveContainer" containerID="44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb"
Jul 29 17:33:34.239302 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.239288    2559 scope.go:117] "RemoveContainer" containerID="483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48"
Jul 29 17:33:34.246448 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.246424    2559 scope.go:117] "RemoveContainer" containerID="756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75"
Jul 29 17:33:34.253505 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.253479    2559 scope.go:117] "RemoveContainer" containerID="44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb"
Jul 29 17:33:34.253824 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:33:34.253801    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb\": container with ID starting with 44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb not found: ID does not exist" containerID="44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb"
Jul 29 17:33:34.254066 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.253835    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb"} err="failed to get container status \"44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb\": rpc error: code = NotFound desc = could not find container \"44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb\": container with ID starting with 44110836386e4ce7c167ae02bea277ea47d75339ef73bb99ee92153a75ddebdb not found: ID does not exist"
Jul 29 17:33:34.254066 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.253856    2559 scope.go:117] "RemoveContainer" containerID="483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48"
Jul 29 17:33:34.254271 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:33:34.254155    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48\": container with ID starting with 483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48 not found: ID does not exist" containerID="483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48"
Jul 29 17:33:34.254271 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.254231    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48"} err="failed to get container status \"483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48\": rpc error: code = NotFound desc = could not find container \"483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48\": container with ID starting with 483d0235e5d8228f5bfa9d8e9a80b65d64cae5ce17d0170ec1ddba32d7452e48 not found: ID does not exist"
Jul 29 17:33:34.254271 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.254254    2559 scope.go:117] "RemoveContainer" containerID="756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75"
Jul 29 17:33:34.254555 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:33:34.254534    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75\": container with ID starting with 756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75 not found: ID does not exist" containerID="756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75"
Jul 29 17:33:34.254595 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.254565    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75"} err="failed to get container status \"756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75\": rpc error: code = NotFound desc = could not find container \"756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75\": container with ID starting with 756ad2ee68bf69867cfc3bd384d18b254c4120f9b2017dfd52f7791f6bfd0e75 not found: ID does not exist"
Jul 29 17:33:34.255132 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.255117    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"]
Jul 29 17:33:34.261072 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.261052    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schecr696"]
Jul 29 17:33:34.703931 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:34.703902    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5114605-a915-4df0-906c-642dbdc35f35" path="/var/lib/kubelet/pods/c5114605-a915-4df0-906c-642dbdc35f35/volumes"
Jul 29 17:33:35.095364 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.095285    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jul 29 17:33:35.095506 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.095475    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" containerName="main" containerID="cri-o://6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de" gracePeriod=30
Jul 29 17:33:35.840312 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.840289    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:33:35.952893 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.952860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4183c4be-6f99-4304-aa37-0e16425653a2-tls-certs\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953060 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.952905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-model-cache\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953060 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvtpz\" (UniqueName: \"kubernetes.io/projected/4183c4be-6f99-4304-aa37-0e16425653a2-kube-api-access-lvtpz\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953203 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953071    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-kserve-provision-location\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953203 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953143    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-tmp-dir\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953203 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953171    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:35.953203 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-dshm\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953413 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953228    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-home\") pod \"4183c4be-6f99-4304-aa37-0e16425653a2\" (UID: \"4183c4be-6f99-4304-aa37-0e16425653a2\") "
Jul 29 17:33:35.953589 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.953553    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-model-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:35.954028 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.954002    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-home" (OuterVolumeSpecName: "home") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:35.955070 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.955047    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4183c4be-6f99-4304-aa37-0e16425653a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:33:35.955250 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.955231    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-dshm" (OuterVolumeSpecName: "dshm") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:35.955468 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.955452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4183c4be-6f99-4304-aa37-0e16425653a2-kube-api-access-lvtpz" (OuterVolumeSpecName: "kube-api-access-lvtpz") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "kube-api-access-lvtpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:33:35.971808 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:35.971783    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:36.010223 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.010074    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4183c4be-6f99-4304-aa37-0e16425653a2" (UID: "4183c4be-6f99-4304-aa37-0e16425653a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:33:36.054498 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.054468    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvtpz\" (UniqueName: \"kubernetes.io/projected/4183c4be-6f99-4304-aa37-0e16425653a2-kube-api-access-lvtpz\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:36.054498 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.054494    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:36.054662 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.054509    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-tmp-dir\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:36.054662 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.054521    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-dshm\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:36.054662 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.054532    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4183c4be-6f99-4304-aa37-0e16425653a2-home\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:36.054662 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.054542    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4183c4be-6f99-4304-aa37-0e16425653a2-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:33:36.240432 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.240363    2559 generic.go:356] "Generic (PLEG): container finished" podID="4183c4be-6f99-4304-aa37-0e16425653a2" containerID="6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de" exitCode=0
Jul 29 17:33:36.240432 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.240413    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jul 29 17:33:36.240432 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.240422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"4183c4be-6f99-4304-aa37-0e16425653a2","Type":"ContainerDied","Data":"6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de"}
Jul 29 17:33:36.240620 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.240463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"4183c4be-6f99-4304-aa37-0e16425653a2","Type":"ContainerDied","Data":"191571ac35e0a475722410b858f209bf8c0dcab8f18a458ebed28401d1df489e"}
Jul 29 17:33:36.240620 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.240493    2559 scope.go:117] "RemoveContainer" containerID="6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de"
Jul 29 17:33:36.249231 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.249214    2559 scope.go:117] "RemoveContainer" containerID="5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce"
Jul 29 17:33:36.262306 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.262286    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jul 29 17:33:36.263360 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.263344    2559 scope.go:117] "RemoveContainer" containerID="6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de"
Jul 29 17:33:36.263606 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:33:36.263589    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de\": container with ID starting with 6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de not found: ID does not exist" containerID="6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de"
Jul 29 17:33:36.263654 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.263615    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de"} err="failed to get container status \"6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de\": rpc error: code = NotFound desc = could not find container \"6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de\": container with ID starting with 6544593c7a7570b27061c7b73915834122b3707525ee026d7a64145f671052de not found: ID does not exist"
Jul 29 17:33:36.263654 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.263632    2559 scope.go:117] "RemoveContainer" containerID="5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce"
Jul 29 17:33:36.263842 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:33:36.263827    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce\": container with ID starting with 5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce not found: ID does not exist" containerID="5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce"
Jul 29 17:33:36.263884 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.263848    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce"} err="failed to get container status \"5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce\": rpc error: code = NotFound desc = could not find container \"5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce\": container with ID starting with 5cb9cd13a4de6baf3424b82bb747d0912d2c581dfb6bcb79c626b052073d4dce not found: ID does not exist"
Jul 29 17:33:36.266624 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.266606    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jul 29 17:33:36.704123 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:33:36.704066    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" path="/var/lib/kubelet/pods/4183c4be-6f99-4304-aa37-0e16425653a2/volumes"
Jul 29 17:34:15.875546 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:15.875512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:34:42.904956 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:42.904917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:34:42.910701 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:42.910669    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"]
Jul 29 17:34:42.910910 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:42.910886    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="main" containerID="cri-o://0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61" gracePeriod=30
Jul 29 17:34:42.910965 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:42.910929    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="tokenizer" containerID="cri-o://57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8" gracePeriod=30
Jul 29 17:34:42.962653 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:42.962613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:34:43.502037 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:43.502004    2559 generic.go:356] "Generic (PLEG): container finished" podID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerID="0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61" exitCode=0
Jul 29 17:34:43.502223 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:43.502098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerDied","Data":"0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61"}
Jul 29 17:34:44.059389 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.059367    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:34:44.169376 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-uds\") pod \"21c84606-1cf2-4a34-b13d-452b6d531ff3\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") "
Jul 29 17:34:44.169376 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-tmp\") pod \"21c84606-1cf2-4a34-b13d-452b6d531ff3\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") "
Jul 29 17:34:44.169578 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169399    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-kserve-provision-location\") pod \"21c84606-1cf2-4a34-b13d-452b6d531ff3\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") "
Jul 29 17:34:44.169578 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169425    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxsxh\" (UniqueName: \"kubernetes.io/projected/21c84606-1cf2-4a34-b13d-452b6d531ff3-kube-api-access-bxsxh\") pod \"21c84606-1cf2-4a34-b13d-452b6d531ff3\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") "
Jul 29 17:34:44.169578 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169453    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-cache\") pod \"21c84606-1cf2-4a34-b13d-452b6d531ff3\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") "
Jul 29 17:34:44.169578 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c84606-1cf2-4a34-b13d-452b6d531ff3-tls-certs\") pod \"21c84606-1cf2-4a34-b13d-452b6d531ff3\" (UID: \"21c84606-1cf2-4a34-b13d-452b6d531ff3\") "
Jul 29 17:34:44.169578 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169561    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "21c84606-1cf2-4a34-b13d-452b6d531ff3" (UID: "21c84606-1cf2-4a34-b13d-452b6d531ff3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:34:44.169839 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169719    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "21c84606-1cf2-4a34-b13d-452b6d531ff3" (UID: "21c84606-1cf2-4a34-b13d-452b6d531ff3"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:34:44.169839 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169775    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-uds\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:34:44.169839 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.169814    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "21c84606-1cf2-4a34-b13d-452b6d531ff3" (UID: "21c84606-1cf2-4a34-b13d-452b6d531ff3"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:34:44.170215 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.170191    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21c84606-1cf2-4a34-b13d-452b6d531ff3" (UID: "21c84606-1cf2-4a34-b13d-452b6d531ff3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:34:44.171565 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.171538    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c84606-1cf2-4a34-b13d-452b6d531ff3-kube-api-access-bxsxh" (OuterVolumeSpecName: "kube-api-access-bxsxh") pod "21c84606-1cf2-4a34-b13d-452b6d531ff3" (UID: "21c84606-1cf2-4a34-b13d-452b6d531ff3"). InnerVolumeSpecName "kube-api-access-bxsxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:34:44.171673 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.171625    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c84606-1cf2-4a34-b13d-452b6d531ff3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21c84606-1cf2-4a34-b13d-452b6d531ff3" (UID: "21c84606-1cf2-4a34-b13d-452b6d531ff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 17:34:44.265809 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.265773    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hzvrh/must-gather-7wctx"]
Jul 29 17:34:44.266057 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266044    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="storage-initializer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266058    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="storage-initializer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266068    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="storage-initializer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266074    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="storage-initializer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266092    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="main"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266098    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="main"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266105    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="tokenizer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266111    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="tokenizer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266118    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="tokenizer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266123    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="tokenizer"
Jul 29 17:34:44.266131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266130    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="storage-initializer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266137    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="storage-initializer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266157    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266162    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266168    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266173    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266178    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="tokenizer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266183    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="tokenizer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266190    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" containerName="storage-initializer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266195    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" containerName="storage-initializer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266249    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="tokenizer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266256    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a70445e-6320-4088-94f8-6fdb9acad359" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266261    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4183c4be-6f99-4304-aa37-0e16425653a2" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266266    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="tokenizer"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266270    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266275    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5114605-a915-4df0-906c-642dbdc35f35" containerName="main"
Jul 29 17:34:44.266415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.266280    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerName="tokenizer"
Jul 29 17:34:44.269322 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.269306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.271005 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.270983    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-tmp\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:34:44.271131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.271007    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-kserve-provision-location\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:34:44.271131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.271023    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsxh\" (UniqueName: \"kubernetes.io/projected/21c84606-1cf2-4a34-b13d-452b6d531ff3-kube-api-access-bxsxh\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:34:44.271131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.271035    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c84606-1cf2-4a34-b13d-452b6d531ff3-tokenizer-cache\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:34:44.271131 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.271047    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c84606-1cf2-4a34-b13d-452b6d531ff3-tls-certs\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:34:44.272102 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.272071    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzvrh\"/\"openshift-service-ca.crt\""
Jul 29 17:34:44.272260 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.272243    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hzvrh\"/\"default-dockercfg-plqhd\""
Jul 29 17:34:44.272352 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.272290    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzvrh\"/\"kube-root-ca.crt\""
Jul 29 17:34:44.278584 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.278565    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzvrh/must-gather-7wctx"]
Jul 29 17:34:44.284151 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.284127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:34:44.347443 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.347415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:34:44.372282 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.372257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d97c1dab-8146-4a9d-a28e-8ab36200bccb-must-gather-output\") pod \"must-gather-7wctx\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") " pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.372379 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.372309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhpx\" (UniqueName: \"kubernetes.io/projected/d97c1dab-8146-4a9d-a28e-8ab36200bccb-kube-api-access-6nhpx\") pod \"must-gather-7wctx\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") " pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.473524 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.473499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d97c1dab-8146-4a9d-a28e-8ab36200bccb-must-gather-output\") pod \"must-gather-7wctx\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") " pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.473650 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.473550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhpx\" (UniqueName: \"kubernetes.io/projected/d97c1dab-8146-4a9d-a28e-8ab36200bccb-kube-api-access-6nhpx\") pod \"must-gather-7wctx\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") " pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.473830 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.473808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d97c1dab-8146-4a9d-a28e-8ab36200bccb-must-gather-output\") pod \"must-gather-7wctx\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") " pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.482199 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.482180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhpx\" (UniqueName: \"kubernetes.io/projected/d97c1dab-8146-4a9d-a28e-8ab36200bccb-kube-api-access-6nhpx\") pod \"must-gather-7wctx\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") " pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.507781 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.507756    2559 generic.go:356] "Generic (PLEG): container finished" podID="21c84606-1cf2-4a34-b13d-452b6d531ff3" containerID="57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8" exitCode=0
Jul 29 17:34:44.507880 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.507789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerDied","Data":"57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8"}
Jul 29 17:34:44.507880 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.507821    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"
Jul 29 17:34:44.507880 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.507829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4" event={"ID":"21c84606-1cf2-4a34-b13d-452b6d531ff3","Type":"ContainerDied","Data":"0f60c2b03a1359714eb9200afd8a0af84cea07d2a3e673490b45b7daa8051d85"}
Jul 29 17:34:44.507880 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.507854    2559 scope.go:117] "RemoveContainer" containerID="57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8"
Jul 29 17:34:44.515642 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.515622    2559 scope.go:117] "RemoveContainer" containerID="0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61"
Jul 29 17:34:44.522705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.522690    2559 scope.go:117] "RemoveContainer" containerID="59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26"
Jul 29 17:34:44.529326 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.529311    2559 scope.go:117] "RemoveContainer" containerID="57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8"
Jul 29 17:34:44.529558 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:34:44.529541    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8\": container with ID starting with 57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8 not found: ID does not exist" containerID="57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8"
Jul 29 17:34:44.529618 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.529564    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8"} err="failed to get container status \"57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8\": rpc error: code = NotFound desc = could not find container \"57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8\": container with ID starting with 57584713855d7afa951218b659b2c1053b6639ce6d2221994b3d18666a9b6ad8 not found: ID does not exist"
Jul 29 17:34:44.529618 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.529579    2559 scope.go:117] "RemoveContainer" containerID="0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61"
Jul 29 17:34:44.529787 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:34:44.529770    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61\": container with ID starting with 0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61 not found: ID does not exist" containerID="0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61"
Jul 29 17:34:44.529827 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.529792    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61"} err="failed to get container status \"0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61\": rpc error: code = NotFound desc = could not find container \"0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61\": container with ID starting with 0dc35e4796246490b9ef9bfc81a0e3fae4b64e1743eaaaeec08f80a86716ce61 not found: ID does not exist"
Jul 29 17:34:44.529827 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.529806    2559 scope.go:117] "RemoveContainer" containerID="59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26"
Jul 29 17:34:44.530050 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:34:44.530032    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26\": container with ID starting with 59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26 not found: ID does not exist" containerID="59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26"
Jul 29 17:34:44.530123 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.530055    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26"} err="failed to get container status \"59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26\": rpc error: code = NotFound desc = could not find container \"59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26\": container with ID starting with 59de09e84a5a98b8aa263faa5fededfb80c9350375d4e93ff4b78c0a1f065a26 not found: ID does not exist"
Jul 29 17:34:44.532717 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.532699    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"]
Jul 29 17:34:44.537991 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.537972    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6bd9fh69w4"]
Jul 29 17:34:44.578425 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.578408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:34:44.702670 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:34:44.702646    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd97c1dab_8146_4a9d_a28e_8ab36200bccb.slice/crio-ac2cd0f6f12e642b91359288b956b420b8a6618fc4668727944a4aef2adfb670 WatchSource:0}: Error finding container ac2cd0f6f12e642b91359288b956b420b8a6618fc4668727944a4aef2adfb670: Status 404 returned error can't find the container with id ac2cd0f6f12e642b91359288b956b420b8a6618fc4668727944a4aef2adfb670
Jul 29 17:34:44.704154 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.704048    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c84606-1cf2-4a34-b13d-452b6d531ff3" path="/var/lib/kubelet/pods/21c84606-1cf2-4a34-b13d-452b6d531ff3/volumes"
Jul 29 17:34:44.704466 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:44.704447    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzvrh/must-gather-7wctx"]
Jul 29 17:34:45.515135 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:45.515099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzvrh/must-gather-7wctx" event={"ID":"d97c1dab-8146-4a9d-a28e-8ab36200bccb","Type":"ContainerStarted","Data":"ac2cd0f6f12e642b91359288b956b420b8a6618fc4668727944a4aef2adfb670"}
Jul 29 17:34:51.544003 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:51.543961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzvrh/must-gather-7wctx" event={"ID":"d97c1dab-8146-4a9d-a28e-8ab36200bccb","Type":"ContainerStarted","Data":"5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb"}
Jul 29 17:34:51.544003 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:51.544008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzvrh/must-gather-7wctx" event={"ID":"d97c1dab-8146-4a9d-a28e-8ab36200bccb","Type":"ContainerStarted","Data":"97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"}
Jul 29 17:34:51.561708 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:51.561628    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hzvrh/must-gather-7wctx" podStartSLOduration=1.6040153780000002 podStartE2EDuration="7.561606436s" totalImagesPullingTime="5.957591058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 17:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:34:51.561175023 +0000 UTC m=+2609.411208723" watchObservedRunningTime="2026-07-29 17:34:51.561606436 +0000 UTC m=+2609.411640136"
Jul 29 17:34:59.773129 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:59.773074    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:34:59.773755 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:34:59.773464    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:00.790522 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:00.790490    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:00.790993 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:00.790937    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:01.832351 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:01.832323    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:01.832935 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:01.832822    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:02.862429 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:02.862402    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:02.862971 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:02.862951    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:03.884807 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:03.884769    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:03.885500 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:03.885474    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:04.911886 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:04.911858    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:04.912365 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:04.912153    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:05.959447 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:05.959419    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:05.959986 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:05.959958    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:06.982982 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:06.982947    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:06.983529 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:06.983491    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:08.002958 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:08.000999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:08.002958 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:08.001312    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:09.025722 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:09.025692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:09.026216 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:09.026130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:10.075367 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:10.075337    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:10.075993 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:10.075950    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:11.149585 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:11.149556    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:11.150210 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:11.150055    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:12.295189 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:12.295127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:12.295693 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:12.295583    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:13.367691 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:13.367657    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/2.log"
Jul 29 17:35:13.368201 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:13.368071    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-579df5d64f-t9b75_581f680c-a2a0-4443-b3ca-4834dfaa9c17/istio-proxy/1.log"
Jul 29 17:35:14.388768 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:14.388716    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/0.log"
Jul 29 17:35:14.389618 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:14.389594    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/1.log"
Jul 29 17:35:15.214425 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:15.214400    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/0.log"
Jul 29 17:35:15.215486 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:15.215464    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/1.log"
Jul 29 17:35:15.995415 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:15.995380    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867c7768fd-xj254_f09bac5c-ea26-48fa-ae61-6f453c4a8506/authorino/0.log"
Jul 29 17:35:16.035989 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:16.035961    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-89xrd_0349333e-24ad-43a2-a6c2-8bf576b49e1c/kuadrant-console-plugin/0.log"
Jul 29 17:35:17.649918 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:17.649885    2559 generic.go:356] "Generic (PLEG): container finished" podID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerID="97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a" exitCode=0
Jul 29 17:35:17.650369 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:17.649956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzvrh/must-gather-7wctx" event={"ID":"d97c1dab-8146-4a9d-a28e-8ab36200bccb","Type":"ContainerDied","Data":"97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"}
Jul 29 17:35:17.650369 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:17.650309    2559 scope.go:117] "RemoveContainer" containerID="97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"
Jul 29 17:35:17.749900 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:17.749873    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hzvrh_must-gather-7wctx_d97c1dab-8146-4a9d-a28e-8ab36200bccb/gather/0.log"
Jul 29 17:35:18.272268 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:18.272157    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:35:19.432732 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:19.432701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:35:21.320871 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:21.320837    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gkqgp_f51ad380-8ca6-47ae-be18-66d23531f26f/global-pull-secret-syncer/0.log"
Jul 29 17:35:21.393503 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:21.393476    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log"
Jul 29 17:35:21.537454 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:21.537431    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-76.ec2.internal_bfca4edb4041480c1eb9593a15543d37/haproxy/0.log"
Jul 29 17:35:23.249394 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.249358    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hzvrh/must-gather-7wctx"]
Jul 29 17:35:23.249855 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.249629    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-hzvrh/must-gather-7wctx" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="copy" containerID="cri-o://5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb" gracePeriod=2
Jul 29 17:35:23.253898 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.253872    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hzvrh/must-gather-7wctx"]
Jul 29 17:35:23.477499 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.477480    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hzvrh_must-gather-7wctx_d97c1dab-8146-4a9d-a28e-8ab36200bccb/copy/0.log"
Jul 29 17:35:23.478969 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.478949    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:35:23.481320 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:35:23.481292    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7wctx\" is forbidden: User \"system:node:ip-10-0-139-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hzvrh\": no relationship found between node 'ip-10-0-139-76.ec2.internal' and this object" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:35:23.499626 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.499581    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d97c1dab-8146-4a9d-a28e-8ab36200bccb-must-gather-output\") pod \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") "
Jul 29 17:35:23.499690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.499628    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nhpx\" (UniqueName: \"kubernetes.io/projected/d97c1dab-8146-4a9d-a28e-8ab36200bccb-kube-api-access-6nhpx\") pod \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\" (UID: \"d97c1dab-8146-4a9d-a28e-8ab36200bccb\") "
Jul 29 17:35:23.501620 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.501596    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97c1dab-8146-4a9d-a28e-8ab36200bccb-kube-api-access-6nhpx" (OuterVolumeSpecName: "kube-api-access-6nhpx") pod "d97c1dab-8146-4a9d-a28e-8ab36200bccb" (UID: "d97c1dab-8146-4a9d-a28e-8ab36200bccb"). InnerVolumeSpecName "kube-api-access-6nhpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 17:35:23.505372 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.505299    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97c1dab-8146-4a9d-a28e-8ab36200bccb-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d97c1dab-8146-4a9d-a28e-8ab36200bccb" (UID: "d97c1dab-8146-4a9d-a28e-8ab36200bccb"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 17:35:23.600134 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.600100    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d97c1dab-8146-4a9d-a28e-8ab36200bccb-must-gather-output\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:35:23.600134 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.600133    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhpx\" (UniqueName: \"kubernetes.io/projected/d97c1dab-8146-4a9d-a28e-8ab36200bccb-kube-api-access-6nhpx\") on node \"ip-10-0-139-76.ec2.internal\" DevicePath \"\""
Jul 29 17:35:23.673023 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.673002    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hzvrh_must-gather-7wctx_d97c1dab-8146-4a9d-a28e-8ab36200bccb/copy/0.log"
Jul 29 17:35:23.673367 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.673345    2559 generic.go:356] "Generic (PLEG): container finished" podID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerID="5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb" exitCode=143
Jul 29 17:35:23.673441 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.673389    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:35:23.673441 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.673437    2559 scope.go:117] "RemoveContainer" containerID="5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb"
Jul 29 17:35:23.676369 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:35:23.676341    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7wctx\" is forbidden: User \"system:node:ip-10-0-139-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hzvrh\": no relationship found between node 'ip-10-0-139-76.ec2.internal' and this object" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:35:23.680881 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.680754    2559 scope.go:117] "RemoveContainer" containerID="97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"
Jul 29 17:35:23.683362 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:35:23.683337    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7wctx\" is forbidden: User \"system:node:ip-10-0-139-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hzvrh\": no relationship found between node 'ip-10-0-139-76.ec2.internal' and this object" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" pod="openshift-must-gather-hzvrh/must-gather-7wctx"
Jul 29 17:35:23.693304 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.693289    2559 scope.go:117] "RemoveContainer" containerID="5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb"
Jul 29 17:35:23.693546 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:35:23.693530    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb\": container with ID starting with 5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb not found: ID does not exist" containerID="5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb"
Jul 29 17:35:23.693609 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.693552    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb"} err="failed to get container status \"5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb\": rpc error: code = NotFound desc = could not find container \"5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb\": container with ID starting with 5a16e5f5ec34f997974c830b3845584c3d4a77485dabd1ee9ed762e3f7ee4aeb not found: ID does not exist"
Jul 29 17:35:23.693609 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.693569    2559 scope.go:117] "RemoveContainer" containerID="97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"
Jul 29 17:35:23.693785 ip-10-0-139-76 kubenswrapper[2559]: E0729 17:35:23.693772    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a\": container with ID starting with 97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a not found: ID does not exist" containerID="97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"
Jul 29 17:35:23.693827 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:23.693790    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a"} err="failed to get container status \"97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a\": rpc error: code = NotFound desc = could not find container \"97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a\": container with ID starting with 97be6512a9bbce05ba852c2e68213eec9b2b0b185f070514966e064bad964a6a not found: ID does not exist"
Jul 29 17:35:24.448794 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:24.448754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:35:24.549243 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:24.549201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmbwc_a9dbe303-c874-4a64-9bb0-b3da59e4ce8f/konnectivity-agent/0.log" limit=1024
Jul 29 17:35:24.703288 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:24.703210    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" path="/var/lib/kubelet/pods/d97c1dab-8146-4a9d-a28e-8ab36200bccb/volumes"
Jul 29 17:35:26.129531 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:26.129501    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867c7768fd-xj254_f09bac5c-ea26-48fa-ae61-6f453c4a8506/authorino/0.log"
Jul 29 17:35:26.218890 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:26.218861    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-89xrd_0349333e-24ad-43a2-a6c2-8bf576b49e1c/kuadrant-console-plugin/0.log"
Jul 29 17:35:28.269753 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:28.269708    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fdstd_7a592f16-4bb0-4558-913c-7a60d49b945d/node-exporter/0.log"
Jul 29 17:35:28.292184 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:28.292160    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fdstd_7a592f16-4bb0-4558-913c-7a60d49b945d/kube-rbac-proxy/0.log"
Jul 29 17:35:28.322150 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:28.322118    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fdstd_7a592f16-4bb0-4558-913c-7a60d49b945d/init-textfile/0.log"
Jul 29 17:35:32.029324 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029295    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"]
Jul 29 17:35:32.029682 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029558    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="copy"
Jul 29 17:35:32.029682 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029569    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="copy"
Jul 29 17:35:32.029682 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029587    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="gather"
Jul 29 17:35:32.029682 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029593    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="gather"
Jul 29 17:35:32.029682 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029649    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="gather"
Jul 29 17:35:32.029682 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.029657    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97c1dab-8146-4a9d-a28e-8ab36200bccb" containerName="copy"
Jul 29 17:35:32.032320 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.032303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.035071 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.035051    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4lvf9\"/\"default-dockercfg-bg4rw\""
Jul 29 17:35:32.036212 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.036195    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lvf9\"/\"kube-root-ca.crt\""
Jul 29 17:35:32.036329 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.036237    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lvf9\"/\"openshift-service-ca.crt\""
Jul 29 17:35:32.044705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.044678    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"]
Jul 29 17:35:32.050690 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.050667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-lib-modules\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.050776 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.050706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-podres\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.050776 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.050747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-sys\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.050861 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.050774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxs6\" (UniqueName: \"kubernetes.io/projected/16589c9e-685f-4989-9109-dd1e3ee02082-kube-api-access-6jxs6\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.050903 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.050848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-proc\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151398 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-lib-modules\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151520 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-podres\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151520 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-sys\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151520 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxs6\" (UniqueName: \"kubernetes.io/projected/16589c9e-685f-4989-9109-dd1e3ee02082-kube-api-access-6jxs6\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151520 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-proc\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151521    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-sys\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-proc\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-lib-modules\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.151665 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.151549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16589c9e-685f-4989-9109-dd1e3ee02082-podres\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.159711 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.159686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxs6\" (UniqueName: \"kubernetes.io/projected/16589c9e-685f-4989-9109-dd1e3ee02082-kube-api-access-6jxs6\") pod \"perf-node-gather-daemonset-pkcht\" (UID: \"16589c9e-685f-4989-9109-dd1e3ee02082\") " pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.342886 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.342811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.466950 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.466926    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"]
Jul 29 17:35:32.469393 ip-10-0-139-76 kubenswrapper[2559]: W0729 17:35:32.469355    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16589c9e_685f_4989_9109_dd1e3ee02082.slice/crio-d2e33b6e31643549be1771fbad0723a7447009d8a961ffcd15436419f0399948 WatchSource:0}: Error finding container d2e33b6e31643549be1771fbad0723a7447009d8a961ffcd15436419f0399948: Status 404 returned error can't find the container with id d2e33b6e31643549be1771fbad0723a7447009d8a961ffcd15436419f0399948
Jul 29 17:35:32.705740 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.705702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht" event={"ID":"16589c9e-685f-4989-9109-dd1e3ee02082","Type":"ContainerStarted","Data":"439b55ae0d8409c9fd9c75c4a8e7723a1fe0dcb27193d901f5c1941710bd501d"}
Jul 29 17:35:32.705905 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.705748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht" event={"ID":"16589c9e-685f-4989-9109-dd1e3ee02082","Type":"ContainerStarted","Data":"d2e33b6e31643549be1771fbad0723a7447009d8a961ffcd15436419f0399948"}
Jul 29 17:35:32.705905 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.705843    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:32.722472 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:32.722424    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht" podStartSLOduration=0.722409765 podStartE2EDuration="722.409765ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 17:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 17:35:32.72135348 +0000 UTC m=+2650.571387178" watchObservedRunningTime="2026-07-29 17:35:32.722409765 +0000 UTC m=+2650.572443497"
Jul 29 17:35:33.370133 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:33.370099    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tbq6b_bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c/dns/0.log"
Jul 29 17:35:33.395184 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:33.395161    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tbq6b_bbb25ba8-1b1b-40cd-9a7a-301beaeecd5c/kube-rbac-proxy/0.log"
Jul 29 17:35:33.445977 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:33.445957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jn7cv_08906a19-3edf-49f2-8a85-6fbf739bbb2e/dns-node-resolver/0.log"
Jul 29 17:35:34.006830 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:34.006795    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56cbcf4895-xtnxt_68555fa0-771f-4771-bc4e-8900251a5071/registry/0.log"
Jul 29 17:35:34.081864 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:34.081833    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvfl4_81cd7021-f3ac-471d-a272-c8ff4492fef5/node-ca/0.log"
Jul 29 17:35:35.107349 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:35.107317    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/0.log"
Jul 29 17:35:35.108367 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:35.108340    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/1.log"
Jul 29 17:35:35.756945 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:35.756919    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q9f27_500cedb2-76db-4404-88b6-21e82828a29c/serve-healthcheck-canary/0.log"
Jul 29 17:35:36.304386 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:36.304351    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6n5vf_c061c174-6b97-4f67-b636-6831a6862ace/kube-rbac-proxy/0.log"
Jul 29 17:35:36.340592 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:36.340568    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6n5vf_c061c174-6b97-4f67-b636-6831a6862ace/exporter/0.log"
Jul 29 17:35:36.388223 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:36.388197    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6n5vf_c061c174-6b97-4f67-b636-6831a6862ace/extractor/0.log"
Jul 29 17:35:38.719384 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:38.719355    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4lvf9/perf-node-gather-daemonset-pkcht"
Jul 29 17:35:39.454072 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:39.454038    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bb667976b-pzw89_95c90d10-8080-4856-8938-6c822078e54f/manager/0.log"
Jul 29 17:35:43.303494 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:43.303461    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/0.log"
Jul 29 17:35:43.304436 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:43.304413    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/1.log"
Jul 29 17:35:44.391469 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:44.391432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/0.log"
Jul 29 17:35:44.392705 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:44.392669    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5c7d4f8778-pfqgs_153d0a1f-89b0-4f64-a1ee-47f3f5c06953/istio-proxy/1.log"
Jul 29 17:35:46.048387 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:46.048354    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dwbc2_458ca5ca-78b2-498f-93df-43015cf80a74/migrator/0.log"
Jul 29 17:35:46.069525 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:46.069504    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dwbc2_458ca5ca-78b2-498f-93df-43015cf80a74/graceful-termination/0.log"
Jul 29 17:35:47.640203 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.640174    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/kube-multus-additional-cni-plugins/0.log"
Jul 29 17:35:47.660114 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.660071    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/egress-router-binary-copy/0.log"
Jul 29 17:35:47.681040 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.681018    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/cni-plugins/0.log"
Jul 29 17:35:47.700413 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.700387    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/bond-cni-plugin/0.log"
Jul 29 17:35:47.721192 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.721166    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/routeoverride-cni/0.log"
Jul 29 17:35:47.741867 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.741841    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/whereabouts-cni-bincopy/0.log"
Jul 29 17:35:47.762093 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:47.762056    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w4tf_ad7706df-e587-4d12-b151-37f895e4eeab/whereabouts-cni/0.log"
Jul 29 17:35:48.207733 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:48.207697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vc2dr_5912cfcc-c880-49bd-90f2-86d4034e5d4d/kube-multus/0.log"
Jul 29 17:35:48.282984 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:48.282947    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-stljd_f1c52131-cd16-415a-a195-0d47aca1a9c4/network-metrics-daemon/0.log"
Jul 29 17:35:48.301383 ip-10-0-139-76 kubenswrapper[2559]: I0729 17:35:48.301354    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-stljd_f1c52131-cd16-415a-a195-0d47aca1a9c4/kube-rbac-proxy/0.log"