Jul 02 13:13:45.138831 ip-10-0-138-135 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.585780    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589855    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589868    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589872    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589875    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589878    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589881    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589883    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:13:45.633103 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589886    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589889    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589892    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589895    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589899    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589905    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589908    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589911    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589914    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589917    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589919    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589922    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589925    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589928    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589930    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589933    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589936    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589939    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589942    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589945    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:13:45.883950 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589947    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589950    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589953    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589955    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589958    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589961    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589963    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589966    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589968    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589971    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589973    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589976    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589979    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589982    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589985    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589988    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589991    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589993    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.589996    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590000    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:13:45.895889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590003    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:13:45.670097 ip-10-0-138-135 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590005    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590010    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590015    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590018    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590021    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590025    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590028    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590031    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590035    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590038    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590041    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590044    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590047    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590049    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590052    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590055    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590058    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590061    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:13:45.897638 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590065    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590069    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590072    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590077    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590079    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590082    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590085    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590088    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590090    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590093    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590095    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590098    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590100    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590103    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590106    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590108    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590111    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590114    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590117    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590120    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:13:45.901105 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590123    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590126    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590128    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590131    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590134    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590137    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590139    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590142    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590144    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590147    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590150    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590153    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590155    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590868    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590875    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590880    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590883    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590887    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590890    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590893    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:13:45.902889 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590896    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590898    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590901    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590904    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590906    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590909    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590911    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590914    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590916    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590919    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590921    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590925    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590928    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590931    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590934    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590936    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590939    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590942    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590944    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:13:45.903941 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590947    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590949    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590952    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590954    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590957    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590960    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590962    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590965    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590967    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590970    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590972    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590975    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590978    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590981    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590986    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590990    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590994    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590996    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.590999    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591002    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:13:45.904562 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591004    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591007    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591009    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591012    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591015    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591017    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591020    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591022    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591025    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591027    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591030    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591033    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591035    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591038    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591041    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591046    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591050    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591053    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591056    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:13:45.905383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591059    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591062    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591065    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591068    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591071    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591073    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591076    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591079    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591082    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591084    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591087    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591089    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591092    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591095    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591097    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591101    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591109    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591111    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591114    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:13:45.905996 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591117    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591119    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591122    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591124    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591127    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591130    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591133    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591135    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591138    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591140    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591143    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591146    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591148    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591151    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591156    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591612    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591620    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591623    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591626    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591630    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:13:45.906626 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591633    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591636    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591639    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591642    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591645    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591649    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591652    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591655    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591657    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591662    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591664    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591667    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591670    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591673    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591675    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591678    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591680    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591683    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591685    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:13:45.907234 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591688    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591691    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591693    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591696    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591698    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591700    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591703    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591707    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591710    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591712    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591715    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591718    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591720    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591723    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591727    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591729    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591732    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591734    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591737    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591739    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:13:45.907842 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591741    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591744    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591748    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591751    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591753    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591756    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591758    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591761    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591763    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591766    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591768    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591771    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591773    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591776    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591778    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591780    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591783    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591785    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591788    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591791    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:13:45.908468 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591794    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591797    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591800    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591802    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591805    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591807    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591811    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591813    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591816    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591818    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591821    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591823    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591826    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591829    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591833    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591836    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591838    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591841    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591844    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591846    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:13:45.909138 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591849    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591852    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591854    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591858    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591861    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591864    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591867    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591870    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591872    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591875    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591878    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591881    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591884    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591886    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.591889    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591951    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591958    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591964    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591969    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591974    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591978    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:13:45.909762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591982    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591986    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591990    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591993    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.591996    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592001    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592004    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592007    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592010    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592014    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592017    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592019    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592024    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592027    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592030    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592034    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592037    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592041    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592044    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592047    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592051    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592054    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592058    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592061    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:13:45.910440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592064    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592067    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592071    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592074    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592077    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592080    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592083    2564 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592091    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592095    2564 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592099    2564 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592102    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592105    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592108    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592113    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592116    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592119    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592122    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592125    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592128    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592130    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592134    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592137    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592140    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592143    2564 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592146    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:13:45.911108 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592149    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592153    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592156    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592159    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592162    2564 flags.go:64] FLAG: --help="false"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592166    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-138-135.ec2.internal"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592169    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592172    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592175    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592179    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592182    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592185    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592188    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592191    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592194    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592197    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592200    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592203    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592206    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592209    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592213    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592216    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592218    2564 flags.go:64] FLAG: --lock-file=""
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592221    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:13:45.911902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592224    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592227    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592233    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592235    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592238    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592241    2564 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592244    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592247    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592250    2564 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592253    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592258    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592261    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592265    2564 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592270    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592273    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592276    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592279    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592282    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592285    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592288    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592295    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592298    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592301    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:13:45.912558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592304    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592307    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592313    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592316    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592319    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592324    2564 flags.go:64] FLAG: --port="10250"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592327    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592330    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0407c4ae3d2939537"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592333    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592336    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592339    2564 flags.go:64] FLAG: --register-node="true"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592342    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592346    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592349    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592352    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592354    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592358    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592361    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592364    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592367    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592370    2564 flags.go:64] FLAG: --runonce="false"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592375    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592381    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592388    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592392    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:13:45.913216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592397    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592401    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592406    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592411    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592416    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592420    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592422    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592425    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592428    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592432    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592435    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592441    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592446    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592449    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592454    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592456    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592459    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592462    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592466    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592469    2564 flags.go:64] FLAG: --v="2"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592473    2564 flags.go:64] FLAG: --version="false"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592477    2564 flags.go:64] FLAG: --vmodule=""
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592481    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592484    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592613    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:13:45.918281 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592618    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592621    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592625    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592628    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592631    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592635    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592638    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592641    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592643    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592646    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592648    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592651    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592654    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592657    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592660    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592662    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592665    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592668    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592670    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592675    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:13:45.919043 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592677    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592680    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592683    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592685    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592691    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592694    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592697    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592699    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592701    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592704    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592707    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592709    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592712    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592714    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592717    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592719    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592722    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592725    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592728    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592731    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:13:46.125829 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592733    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592736    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592739    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592742    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592744    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592747    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592749    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592752    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592754    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592757    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592760    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592764    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592767    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592769    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592772    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592774    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592777    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592779    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592782    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:13:46.126466 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592784    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592787    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592791    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592794    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592797    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592800    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592803    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592806    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592809    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592813    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592818    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592821    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592823    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592826    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592828    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592831    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592834    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592838    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592841    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:13:46.127005 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592844    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592846    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592849    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592852    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592855    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592860    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592863    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592865    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592868    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592871    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592873    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592875    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592878    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592880    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592883    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592885    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592888    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592891    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592893    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:13:46.127557 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.592896    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.592902    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.603108    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.603123    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.603145    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.603154    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603217    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603223    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603226    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603230    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603233    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603236    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603239    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603242    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603244    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:13:46.128064 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603247    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603250    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603253    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603256    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603259    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603262    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603264    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603267    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603270    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603272    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603277    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603280    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603283    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603285    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603288    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603291    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603294    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603297    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603300    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603302    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:13:46.128510 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603308    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603312    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603315    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603318    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603321    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603323    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603326    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603329    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603332    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603336    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603339    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603342    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603345    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603347    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603350    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603352    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603355    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603358    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603361    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:13:46.129072 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603363    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603366    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603368    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603374    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603376    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603379    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603381    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603384    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603386    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603389    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603391    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603394    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603396    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603400    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603403    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603405    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603408    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603410    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603413    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:13:46.129639 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603415    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603418    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603420    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603423    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603425    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603428    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603430    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603432    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603435    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603438    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603440    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603443    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603446    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603448    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603451    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603453    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603457    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603459    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603462    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603465    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:13:46.130146 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603467    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603470    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603472    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603476    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603479    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603482    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603486    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603489    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603491    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603494    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603497    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603500    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.603505    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603638    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603644    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:13:46.130723 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603647    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603650    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603654    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603658    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603661    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603664    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603667    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603670    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603673    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603676    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603679    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603682    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603684    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603688    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603693    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603696    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603699    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603702    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603705    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603708    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:13:46.131137 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603711    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603714    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603716    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603721    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603724    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603727    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603730    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603733    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603736    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603738    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603741    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603744    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603747    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603749    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603752    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603755    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603757    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603760    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603762    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603765    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:13:46.131705 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603767    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603770    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603773    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603776    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603778    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603781    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603784    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603787    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603791    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603794    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603797    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603800    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603803    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603806    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603808    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603813    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603816    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603819    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603821    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:13:46.132262 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603824    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603826    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603829    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603831    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603834    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603836    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603839    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603841    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603844    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603847    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603849    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603852    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603854    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603857    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603860    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603862    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603865    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603867    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603870    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603874    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:13:46.132831 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603876    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603879    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603881    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603884    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603886    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603889    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603891    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603894    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603898    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603901    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603903    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603906    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603908    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603911    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603913    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603916    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603918    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:13:46.133383 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:45.603920    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.603925    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.604030    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.606472    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.608317    2564 server.go:1019] "Starting client certificate rotation"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.608424    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.608491    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.611001    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.617668    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.629966    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.636459    2564 log.go:25] "Validated CRI v1 image API"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.637248    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:13:46.133898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.639775    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7f4caac5-ea21-46b8-ac34-e1d47ea772f6:/dev/nvme0n1p3 ac68b4e6-5f5a-4c7c-b836-775875e70fdd:/dev/nvme0n1p4]
Jul 02 13:13:46.134280 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.639790    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.645762    2564 manager.go:216] Machine: {Timestamp:2026-07-02 13:13:45.644314435 +0000 UTC m=+0.375352577 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108266 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2424c0def9144d01115219cb504c9b SystemUUID:ec2424c0-def9-144d-0111-5219cb504c9b BootID:ec686609-08ed-4bda-9558-bb90ee360676 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:07:64:30:0d:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:07:64:30:0d:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:89:49:af:55:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.645862    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.645976    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.646581    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.649454    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.649485    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-135.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:13:46.134421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.650343    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.650353    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.650360    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.650375    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.651708    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656141    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656252    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656295    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656310    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656325    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656334    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656344    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656803    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.656824    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.659171    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.663634    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.663684    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664495    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664509    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664516    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664545    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664560    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664569    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664578    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664584    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664592    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664599    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664608    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.664614    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.665636    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.665665    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.668579    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.668658    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-135.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.668690    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-135.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.669166    2564 server.go:1268] "Started kubelet"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.669255    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.669309    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.669398    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.670881    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:13:46.134707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.671367    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.674770    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.678016    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-135.ec2.internal.18be7b41f33b4718  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-135.ec2.internal,UID:ip-10-0-138-135.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-135.ec2.internal,},FirstTimestamp:2026-07-02 13:13:45.669134104 +0000 UTC m=+0.400172225,LastTimestamp:2026-07-02 13:13:45.669134104 +0000 UTC m=+0.400172225,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-135.ec2.internal,}"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.679167    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.679187    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.679267    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.679959    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.680028    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.680043    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.680144    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.680150    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.681762    2564 factory.go:55] Registering systemd factory
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.681782    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.681985    2564 factory.go:153] Registering CRI-O factory
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.681999    2564 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.682041    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.682069    2564 factory.go:103] Registering Raw factory
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.682091    2564 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.682506    2564 manager.go:349] Starting recovery of all containers
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.686045    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.686090    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-135.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.694228    2564 manager.go:354] Recovery completed
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.699004    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.700981    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.701018    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.701033    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.701659    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.701669    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.701670    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.701711    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.702432    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.702456    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.135751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.702465    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.703298    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-135.ec2.internal.18be7b41f521804f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-135.ec2.internal,UID:ip-10-0-138-135.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-135.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-135.ec2.internal,},FirstTimestamp:2026-07-02 13:13:45.700999247 +0000 UTC m=+0.432037370,LastTimestamp:2026-07-02 13:13:45.700999247 +0000 UTC m=+0.432037370,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-135.ec2.internal,}"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.703806    2564 policy_none.go:49] "None policy: Start"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.703823    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.703835    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.705840    2564 policy_none.go:47] "Start"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.707892    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2mnw"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.714095    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-135.ec2.internal.18be7b41f521e808  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-135.ec2.internal,UID:ip-10-0-138-135.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-135.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-135.ec2.internal,},FirstTimestamp:2026-07-02 13:13:45.7010258 +0000 UTC m=+0.432063921,LastTimestamp:2026-07-02 13:13:45.7010258 +0000 UTC m=+0.432063921,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-135.ec2.internal,}"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.716285    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2mnw"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.723160    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-135.ec2.internal.18be7b41f5221128  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-135.ec2.internal,UID:ip-10-0-138-135.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-135.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-135.ec2.internal,},FirstTimestamp:2026-07-02 13:13:45.701036328 +0000 UTC m=+0.432074448,LastTimestamp:2026-07-02 13:13:45.701036328 +0000 UTC m=+0.432074448,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-135.ec2.internal,}"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743183    2564 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.743229    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743244    2564 server.go:87] "Starting device plugin registration server"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743538    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743552    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743660    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743739    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.743748    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.744157    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.744208    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.136958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.752674    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.753772    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.753809    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.753831    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.753866    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.756289    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.844142    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.845207    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.845241    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.845251    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.845277    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.854029    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"]
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.854088    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.854951    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.854977    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.854989    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.855002    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.855020    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-135.ec2.internal\": node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.857125    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.857248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.857278    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.858080    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.858100    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.858113    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.858149    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.858176    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.858190    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.860757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.860781    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.861474    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.861501    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.861557    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.140023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.881019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3b8679aa0e9fabe089c7dff63237f97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal\" (UID: \"e3b8679aa0e9fabe089c7dff63237f97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.881042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3b8679aa0e9fabe089c7dff63237f97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal\" (UID: \"e3b8679aa0e9fabe089c7dff63237f97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.881059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7a4c8ad16d6f669b998210c1cb6767-config\") pod \"kube-apiserver-proxy-ip-10-0-138-135.ec2.internal\" (UID: \"eb7a4c8ad16d6f669b998210c1cb6767\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.888644    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.896297    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-135.ec2.internal\" not found" node="ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.901109    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-135.ec2.internal\" not found" node="ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.981589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3b8679aa0e9fabe089c7dff63237f97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal\" (UID: \"e3b8679aa0e9fabe089c7dff63237f97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.981617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3b8679aa0e9fabe089c7dff63237f97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal\" (UID: \"e3b8679aa0e9fabe089c7dff63237f97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.981633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7a4c8ad16d6f669b998210c1cb6767-config\") pod \"kube-apiserver-proxy-ip-10-0-138-135.ec2.internal\" (UID: \"eb7a4c8ad16d6f669b998210c1cb6767\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.981662    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7a4c8ad16d6f669b998210c1cb6767-config\") pod \"kube-apiserver-proxy-ip-10-0-138-135.ec2.internal\" (UID: \"eb7a4c8ad16d6f669b998210c1cb6767\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.981678    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3b8679aa0e9fabe089c7dff63237f97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal\" (UID: \"e3b8679aa0e9fabe089c7dff63237f97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:45.981685    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3b8679aa0e9fabe089c7dff63237f97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal\" (UID: \"e3b8679aa0e9fabe089c7dff63237f97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:45.989711    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.090034    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.190875    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.201049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.287016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.206230    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:46.486100 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.291693    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.486100 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.392260    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.677635 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.492785    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.677635 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.593740    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.677635 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.606829    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:13:46.677635 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.606984    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.693822    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.723298    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:08:46 +0000 UTC" deadline="2028-03-18 14:48:19.019363416 +0000 UTC"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.723330    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15001h34m32.296036201s"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.775799    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.776751    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.776779    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.776791    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:13:46.890197 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.794286    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:46.895310 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.895045    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:47.026843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:46.988119    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:13:47.026843 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:46.996179    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-135.ec2.internal\" not found"
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.050826    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.079607    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.079701    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.095487    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.096704    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal"
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:47.097644    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7a4c8ad16d6f669b998210c1cb6767.slice/crio-7ef9aa18ab294903c7b64500081176c8221603b1abb1f8ffb69cc451784ac537 WatchSource:0}: Error finding container 7ef9aa18ab294903c7b64500081176c8221603b1abb1f8ffb69cc451784ac537: Status 404 returned error can't find the container with id 7ef9aa18ab294903c7b64500081176c8221603b1abb1f8ffb69cc451784ac537
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:47.098089    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b8679aa0e9fabe089c7dff63237f97.slice/crio-85d009ff4da026cc86d790a79503f4590cdcf95effb7787b86c39a40c44fa487 WatchSource:0}: Error finding container 85d009ff4da026cc86d790a79503f4590cdcf95effb7787b86c39a40c44fa487: Status 404 returned error can't find the container with id 85d009ff4da026cc86d790a79503f4590cdcf95effb7787b86c39a40c44fa487
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.103156    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:13:47.176162 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.109097    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:13:47.674919 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.667911    2564 apiserver.go:52] "Watching apiserver"
Jul 02 13:13:47.680985 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.680522    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.684559    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.686018    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wddlr","kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2","openshift-cluster-node-tuning-operator/tuned-p55mb","openshift-image-registry/node-ca-klrnt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal","openshift-multus/network-metrics-daemon-q86fn","openshift-network-operator/iptables-alerter-588qg","kube-system/global-pull-secret-syncer-kkm78","openshift-dns/node-resolver-8qzd5","openshift-multus/multus-224qj","openshift-multus/multus-additional-cni-plugins-6bdbq","openshift-network-diagnostics/network-check-target-8cn72","openshift-ovn-kubernetes/ovnkube-node-cfn87"]
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.687248    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.688802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.688876    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.691125    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.691211    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.693945    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.695283    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.696282    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.697059    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.697496    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.697574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z4w8t\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.697622    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.699146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.699423    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.699684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.700212    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p6cw6\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.701789    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.702375    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.702463    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.702725    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49kxc\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.702869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.830170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.704344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.704403    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nfq5w\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.704415    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.705710    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.705737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.705811    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.705850    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.705944    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.706100    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.706825    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.709379    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7rf9g\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.709663    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.710191    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.710329    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.712616    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.714123    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.714206    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.714372    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sb956\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.714376    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.715265    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b86vc\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.715308    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.715528    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.716220    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.718474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.718736    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.719037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.719046    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.719109    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.719179    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-47nlq\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.720962    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.721154    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:13:47.831838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.721434    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vvjkt\""
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.745621    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7ndlf"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.754783    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7ndlf"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.759781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal" event={"ID":"eb7a4c8ad16d6f669b998210c1cb6767","Type":"ContainerStarted","Data":"7ef9aa18ab294903c7b64500081176c8221603b1abb1f8ffb69cc451784ac537"}
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.762121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal" event={"ID":"e3b8679aa0e9fabe089c7dff63237f97","Type":"ContainerStarted","Data":"85d009ff4da026cc86d790a79503f4590cdcf95effb7787b86c39a40c44fa487"}
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.781579    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.796980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-hostroot\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-slash\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-run-netns\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-system-cni-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-k8s-cni-cncf-io\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-kubelet\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-etc-kubernetes\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-node-log\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-cni-netd\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.833173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-device-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797275    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysconfig\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-run\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/189b2773-1cfd-4c74-89a5-7387b4f5adca-host-slash\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-os-release\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-modprobe-d\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysctl-conf\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-sys\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-ovn\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-log-socket\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-cni-multus\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-registration-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwhr\" (UniqueName: \"kubernetes.io/projected/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-kube-api-access-zvwhr\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-lib-modules\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-kubelet\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-systemd-units\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-env-overrides\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-conf-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-sys-fs\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-systemd\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-var-lib-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.834724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cnibin\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-socket-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-cni-bin\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovnkube-config\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48th\" (UniqueName: \"kubernetes.io/projected/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-kube-api-access-b48th\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-netns\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.797962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-cni-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-os-release\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7133a6f-870e-4461-9c40-37e238ac59a2-serviceca\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-cni-bin\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4rsx\" (UniqueName: \"kubernetes.io/projected/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-kube-api-access-x4rsx\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-etc-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.835405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovn-node-metrics-cert\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktnf\" (UniqueName: \"kubernetes.io/projected/f11610b8-2546-4823-88c3-8d6564b1aac8-kube-api-access-vktnf\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-tuned\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798177    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-system-cni-dir\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysctl-d\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/189b2773-1cfd-4c74-89a5-7387b4f5adca-iptables-alerter-script\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7133a6f-870e-4461-9c40-37e238ac59a2-host\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-cnibin\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-cni-binary-copy\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-socket-dir-parent\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-daemon-config\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-host\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48bn\" (UniqueName: \"kubernetes.io/projected/e7133a6f-870e-4461-9c40-37e238ac59a2-kube-api-access-g48bn\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.836374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554f3e5-e6ba-4f9e-8211-c43c57869d19-tmp-dir\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-systemd\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvfn\" (UniqueName: \"kubernetes.io/projected/ce7f1930-b1ee-41b9-a3a9-076991fdd836-kube-api-access-6fvfn\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03505741-d6c7-4043-b2b0-28878d7c2a5d-dbus\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1554f3e5-e6ba-4f9e-8211-c43c57869d19-hosts-file\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbbbf\" (UniqueName: \"kubernetes.io/projected/1554f3e5-e6ba-4f9e-8211-c43c57869d19-kube-api-access-nbbbf\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86h8n\" (UniqueName: \"kubernetes.io/projected/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-kube-api-access-86h8n\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-var-lib-kubelet\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdsx\" (UniqueName: \"kubernetes.io/projected/189b2773-1cfd-4c74-89a5-7387b4f5adca-kube-api-access-7gdsx\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-multus-certs\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-etc-selinux\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.837097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-kubernetes\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.837781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7f1930-b1ee-41b9-a3a9-076991fdd836-tmp\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.837781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovnkube-script-lib\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.837781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8369f04-06c1-484f-b940-555722659bf6-agent-certs\") pod \"konnectivity-agent-wddlr\" (UID: \"f8369f04-06c1-484f-b940-555722659bf6\") " pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.837781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8369f04-06c1-484f-b940-555722659bf6-konnectivity-ca\") pod \"konnectivity-agent-wddlr\" (UID: \"f8369f04-06c1-484f-b940-555722659bf6\") " pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.837781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.798931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03505741-d6c7-4043-b2b0-28878d7c2a5d-kubelet-config\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-cni-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-os-release\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7133a6f-870e-4461-9c40-37e238ac59a2-serviceca\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-cni-bin\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4rsx\" (UniqueName: \"kubernetes.io/projected/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-kube-api-access-x4rsx\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-etc-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899502    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-cni-bin\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovn-node-metrics-cert\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899611    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-etc-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktnf\" (UniqueName: \"kubernetes.io/projected/f11610b8-2546-4823-88c3-8d6564b1aac8-kube-api-access-vktnf\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899706    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-cni-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-tuned\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-system-cni-dir\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.927215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysctl-d\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/189b2773-1cfd-4c74-89a5-7387b4f5adca-iptables-alerter-script\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899847    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-system-cni-dir\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7133a6f-870e-4461-9c40-37e238ac59a2-host\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-cnibin\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-cni-binary-copy\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-socket-dir-parent\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-daemon-config\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899979    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7133a6f-870e-4461-9c40-37e238ac59a2-host\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899979    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900026    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7133a6f-870e-4461-9c40-37e238ac59a2-serviceca\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900060    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-cnibin\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899998    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysctl-d\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900065    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-socket-dir-parent\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.899998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-host\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.929636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48bn\" (UniqueName: \"kubernetes.io/projected/e7133a6f-870e-4461-9c40-37e238ac59a2-kube-api-access-g48bn\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554f3e5-e6ba-4f9e-8211-c43c57869d19-tmp-dir\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-systemd\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvfn\" (UniqueName: \"kubernetes.io/projected/ce7f1930-b1ee-41b9-a3a9-076991fdd836-kube-api-access-6fvfn\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03505741-d6c7-4043-b2b0-28878d7c2a5d-dbus\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1554f3e5-e6ba-4f9e-8211-c43c57869d19-hosts-file\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbbbf\" (UniqueName: \"kubernetes.io/projected/1554f3e5-e6ba-4f9e-8211-c43c57869d19-kube-api-access-nbbbf\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86h8n\" (UniqueName: \"kubernetes.io/projected/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-kube-api-access-86h8n\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-var-lib-kubelet\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdsx\" (UniqueName: \"kubernetes.io/projected/189b2773-1cfd-4c74-89a5-7387b4f5adca-kube-api-access-7gdsx\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900408    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-multus-certs\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.930304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-etc-selinux\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-kubernetes\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900568    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/189b2773-1cfd-4c74-89a5-7387b4f5adca-iptables-alerter-script\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7f1930-b1ee-41b9-a3a9-076991fdd836-tmp\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900618    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-cni-binary-copy\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovnkube-script-lib\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8369f04-06c1-484f-b940-555722659bf6-agent-certs\") pod \"konnectivity-agent-wddlr\" (UID: \"f8369f04-06c1-484f-b940-555722659bf6\") " pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8369f04-06c1-484f-b940-555722659bf6-konnectivity-ca\") pod \"konnectivity-agent-wddlr\" (UID: \"f8369f04-06c1-484f-b940-555722659bf6\") " pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900703    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-daemon-config\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03505741-d6c7-4043-b2b0-28878d7c2a5d-kubelet-config\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-hostroot\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-slash\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900571    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-os-release\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-run-netns\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900700    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03505741-d6c7-4043-b2b0-28878d7c2a5d-dbus\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.930994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.900996    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554f3e5-e6ba-4f9e-8211-c43c57869d19-tmp-dir\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901105    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-host\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-system-cni-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901126    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-slash\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-k8s-cni-cncf-io\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-kubelet\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901176    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-run-netns\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-etc-kubernetes\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-node-log\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-cni-netd\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901320    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03505741-d6c7-4043-b2b0-28878d7c2a5d-kubelet-config\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-device-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.931596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysconfig\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-run\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/189b2773-1cfd-4c74-89a5-7387b4f5adca-host-slash\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901554    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-systemd\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-os-release\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-modprobe-d\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysctl-conf\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-sys\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.901727    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901502    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-etc-selinux\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901780    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8369f04-06c1-484f-b940-555722659bf6-konnectivity-ca\") pod \"konnectivity-agent-wddlr\" (UID: \"f8369f04-06c1-484f-b940-555722659bf6\") " pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-ovn\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901801    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-kubernetes\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901800    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovnkube-script-lib\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901861    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-hostroot\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.932380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901862    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-system-cni-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901854    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-sys\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-log-socket\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901941    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-node-log\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-cni-multus\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.901978    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-cni-netd\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902050    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-var-lib-kubelet\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902140    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-multus-certs\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902147    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-registration-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwhr\" (UniqueName: \"kubernetes.io/projected/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-kube-api-access-zvwhr\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902192    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-k8s-cni-cncf-io\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-lib-modules\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.933104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902233    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-kubelet\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902259    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1554f3e5-e6ba-4f9e-8211-c43c57869d19-hosts-file\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-kubelet\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902273    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-etc-kubernetes\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902302    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902365    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-ovn\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902377    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902398    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-os-release\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902457    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-lib-modules\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902511    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-kubelet\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902549    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902568    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-run\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902628    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-var-lib-cni-multus\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902656    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-modprobe-d\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.935111 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.902674    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:48.402635539 +0000 UTC m=+3.133673669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902733    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/189b2773-1cfd-4c74-89a5-7387b4f5adca-host-slash\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902748    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysctl-conf\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902783    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-registration-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-systemd-units\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902836    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-log-socket\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902838    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-env-overrides\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902927    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-device-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903006    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-conf-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.902996    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-sysconfig\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903026    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903073    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-multus-conf-dir\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.935976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903082    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-systemd-units\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-sys-fs\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-systemd\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-var-lib-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cnibin\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903245    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-sys-fs\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-socket-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903300    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-run-systemd\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903300    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-var-lib-openvswitch\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-cni-bin\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovnkube-config\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903432    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-socket-dir\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48th\" (UniqueName: \"kubernetes.io/projected/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-kube-api-access-b48th\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903490    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-cnibin\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.937951 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-netns\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903541    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-env-overrides\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903709    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-cni-bin\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903735    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-host-run-netns\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.903788    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.903930    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:13:48.40390101 +0000 UTC m=+3.134939119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.903970    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.904090    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.904441    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovn-node-metrics-cert\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.904485    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-ovnkube-config\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.904623    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8369f04-06c1-484f-b940-555722659bf6-agent-certs\") pod \"konnectivity-agent-wddlr\" (UID: \"f8369f04-06c1-484f-b940-555722659bf6\") " pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.904835    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7f1930-b1ee-41b9-a3a9-076991fdd836-tmp\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.904888    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce7f1930-b1ee-41b9-a3a9-076991fdd836-etc-tuned\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:47.938586 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.906945    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.906962    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.906973    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:47.907555    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:48.407534342 +0000 UTC m=+3.138572473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.909024    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4rsx\" (UniqueName: \"kubernetes.io/projected/bb3278bb-67d1-4e15-a8d6-30b5b204ddcb-kube-api-access-x4rsx\") pod \"multus-224qj\" (UID: \"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb\") " pod="openshift-multus/multus-224qj"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.910034    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdsx\" (UniqueName: \"kubernetes.io/projected/189b2773-1cfd-4c74-89a5-7387b4f5adca-kube-api-access-7gdsx\") pod \"iptables-alerter-588qg\" (UID: \"189b2773-1cfd-4c74-89a5-7387b4f5adca\") " pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.910073    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktnf\" (UniqueName: \"kubernetes.io/projected/f11610b8-2546-4823-88c3-8d6564b1aac8-kube-api-access-vktnf\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.910484    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvfn\" (UniqueName: \"kubernetes.io/projected/ce7f1930-b1ee-41b9-a3a9-076991fdd836-kube-api-access-6fvfn\") pod \"tuned-p55mb\" (UID: \"ce7f1930-b1ee-41b9-a3a9-076991fdd836\") " pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.912039    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbbbf\" (UniqueName: \"kubernetes.io/projected/1554f3e5-e6ba-4f9e-8211-c43c57869d19-kube-api-access-nbbbf\") pod \"node-resolver-8qzd5\" (UID: \"1554f3e5-e6ba-4f9e-8211-c43c57869d19\") " pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.913126    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86h8n\" (UniqueName: \"kubernetes.io/projected/908c8eb7-5b2b-49bd-8952-b68e85db9c4c-kube-api-access-86h8n\") pod \"multus-additional-cni-plugins-6bdbq\" (UID: \"908c8eb7-5b2b-49bd-8952-b68e85db9c4c\") " pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.916663    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48bn\" (UniqueName: \"kubernetes.io/projected/e7133a6f-870e-4461-9c40-37e238ac59a2-kube-api-access-g48bn\") pod \"node-ca-klrnt\" (UID: \"e7133a6f-870e-4461-9c40-37e238ac59a2\") " pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.917322    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwhr\" (UniqueName: \"kubernetes.io/projected/e0975e29-ddef-4a4b-ac06-d29c0f1cbb59-kube-api-access-zvwhr\") pod \"aws-ebs-csi-driver-node-fzlm2\" (UID: \"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:47.917375    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48th\" (UniqueName: \"kubernetes.io/projected/e3d20200-1c44-48fd-80ad-345e9bc3d8b5-kube-api-access-b48th\") pod \"ovnkube-node-cfn87\" (UID: \"e3d20200-1c44-48fd-80ad-345e9bc3d8b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.011483    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.020848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:13:48.056528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.028917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p55mb"
Jul 02 13:13:48.059184 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.029852    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0975e29_ddef_4a4b_ac06_d29c0f1cbb59.slice/crio-7b3c82d1aed48bfb669d382f741c7c35ded47ed884ebda119844cbea40907b54 WatchSource:0}: Error finding container 7b3c82d1aed48bfb669d382f741c7c35ded47ed884ebda119844cbea40907b54: Status 404 returned error can't find the container with id 7b3c82d1aed48bfb669d382f741c7c35ded47ed884ebda119844cbea40907b54
Jul 02 13:13:48.059184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.037432    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-klrnt"
Jul 02 13:13:48.059184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.044759    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:13:48.059184 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.048663    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7f1930_b1ee_41b9_a3a9_076991fdd836.slice/crio-e4be14701d2cb0ffa622444d4265bbf5a7e4581a94025f912e5ce0488f61d7d4 WatchSource:0}: Error finding container e4be14701d2cb0ffa622444d4265bbf5a7e4581a94025f912e5ce0488f61d7d4: Status 404 returned error can't find the container with id e4be14701d2cb0ffa622444d4265bbf5a7e4581a94025f912e5ce0488f61d7d4
Jul 02 13:13:48.059184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.056357    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-588qg"
Jul 02 13:13:48.063528 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.062811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8qzd5"
Jul 02 13:13:48.063528 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.062958    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7133a6f_870e_4461_9c40_37e238ac59a2.slice/crio-c4a0c4e30dacff8274833f96381badba9091329b89e29052e936e99ce8d99795 WatchSource:0}: Error finding container c4a0c4e30dacff8274833f96381badba9091329b89e29052e936e99ce8d99795: Status 404 returned error can't find the container with id c4a0c4e30dacff8274833f96381badba9091329b89e29052e936e99ce8d99795
Jul 02 13:13:48.150593 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.063923    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8369f04_06c1_484f_b940_555722659bf6.slice/crio-8a09d04ada296f59539db5153b3a92778659175a0ed569374800d47d9b982240 WatchSource:0}: Error finding container 8a09d04ada296f59539db5153b3a92778659175a0ed569374800d47d9b982240: Status 404 returned error can't find the container with id 8a09d04ada296f59539db5153b3a92778659175a0ed569374800d47d9b982240
Jul 02 13:13:48.150593 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.069054    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189b2773_1cfd_4c74_89a5_7387b4f5adca.slice/crio-4ba89300ef23e30f3df4cf5f88224755b1991d91af517f67ec4bf3dc6865ad67 WatchSource:0}: Error finding container 4ba89300ef23e30f3df4cf5f88224755b1991d91af517f67ec4bf3dc6865ad67: Status 404 returned error can't find the container with id 4ba89300ef23e30f3df4cf5f88224755b1991d91af517f67ec4bf3dc6865ad67
Jul 02 13:13:48.150593 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.073944    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-224qj"
Jul 02 13:13:48.150593 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.076141    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bdbq"
Jul 02 13:13:48.150593 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.077150    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1554f3e5_e6ba_4f9e_8211_c43c57869d19.slice/crio-c75026de3d3b5a3ce07ff9419ca72a5b609f9d36c676017943de6e3b90425d9f WatchSource:0}: Error finding container c75026de3d3b5a3ce07ff9419ca72a5b609f9d36c676017943de6e3b90425d9f: Status 404 returned error can't find the container with id c75026de3d3b5a3ce07ff9419ca72a5b609f9d36c676017943de6e3b90425d9f
Jul 02 13:13:48.150593 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:13:48.094178    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908c8eb7_5b2b_49bd_8952_b68e85db9c4c.slice/crio-42fc9d42609ff0adcb1a77f6e218005f651ce1ddbf300d8f26bcbbbb2a8578cd WatchSource:0}: Error finding container 42fc9d42609ff0adcb1a77f6e218005f651ce1ddbf300d8f26bcbbbb2a8578cd: Status 404 returned error can't find the container with id 42fc9d42609ff0adcb1a77f6e218005f651ce1ddbf300d8f26bcbbbb2a8578cd
Jul 02 13:13:48.407793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.407694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:48.407793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.407762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.407800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.407887    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.407945    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.407966    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.407974    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:13:49.407952281 +0000 UTC m=+4.138990395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.407984    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.407998    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:48.408014 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.408006    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:49.407988626 +0000 UTC m=+4.139026739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:48.408343 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:48.408047    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:49.408031629 +0000 UTC m=+4.139069744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:48.758636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.758459    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:08:47 +0000 UTC" deadline="2028-03-16 09:59:22.46900278 +0000 UTC"
Jul 02 13:13:48.758636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.758500    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14948h45m33.710507864s"
Jul 02 13:13:48.766039 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.765925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p55mb" event={"ID":"ce7f1930-b1ee-41b9-a3a9-076991fdd836","Type":"ContainerStarted","Data":"e4be14701d2cb0ffa622444d4265bbf5a7e4581a94025f912e5ce0488f61d7d4"}
Jul 02 13:13:48.767616 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.767581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-224qj" event={"ID":"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb","Type":"ContainerStarted","Data":"dac50c49bfb5d0efa4944216deab3eddad02ab9b86d2848b2c9cdb90a60e5431"}
Jul 02 13:13:48.769625 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.769551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-588qg" event={"ID":"189b2773-1cfd-4c74-89a5-7387b4f5adca","Type":"ContainerStarted","Data":"4ba89300ef23e30f3df4cf5f88224755b1991d91af517f67ec4bf3dc6865ad67"}
Jul 02 13:13:48.770987 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.770966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-klrnt" event={"ID":"e7133a6f-870e-4461-9c40-37e238ac59a2","Type":"ContainerStarted","Data":"c4a0c4e30dacff8274833f96381badba9091329b89e29052e936e99ce8d99795"}
Jul 02 13:13:48.773262 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.773235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"eef41e5d0f35cd101f1c40b1d9c48106b69b3cae7cbab4afc6b74181ab97a349"}
Jul 02 13:13:48.774992 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.774938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2" event={"ID":"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59","Type":"ContainerStarted","Data":"7b3c82d1aed48bfb669d382f741c7c35ded47ed884ebda119844cbea40907b54"}
Jul 02 13:13:48.776956 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.776929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerStarted","Data":"42fc9d42609ff0adcb1a77f6e218005f651ce1ddbf300d8f26bcbbbb2a8578cd"}
Jul 02 13:13:48.778110 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.778084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8qzd5" event={"ID":"1554f3e5-e6ba-4f9e-8211-c43c57869d19","Type":"ContainerStarted","Data":"c75026de3d3b5a3ce07ff9419ca72a5b609f9d36c676017943de6e3b90425d9f"}
Jul 02 13:13:48.780454 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:48.780428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wddlr" event={"ID":"f8369f04-06c1-484f-b940-555722659bf6","Type":"ContainerStarted","Data":"8a09d04ada296f59539db5153b3a92778659175a0ed569374800d47d9b982240"}
Jul 02 13:13:49.417720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.417671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:49.417923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.417747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:49.417923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.417777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:49.418109 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.417963    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:13:49.418109 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.417981    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:13:49.418109 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.417995    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:49.418109 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.418059    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:51.418039491 +0000 UTC m=+6.149077606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:49.418521 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.418498    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:49.418635 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.418561    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:13:51.418543968 +0000 UTC m=+6.149582081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:49.418701 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.418639    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:49.418701 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.418673    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:51.418661731 +0000 UTC m=+6.149699845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:49.756285 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.756055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:49.756285 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.756215    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:49.756534 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.756433    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:49.756659 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.756628    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:49.756733 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.756722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:49.756836 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:49.756799    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:13:49.761115 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.761045    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:08:47 +0000 UTC" deadline="2027-12-25 22:56:06.986877216 +0000 UTC"
Jul 02 13:13:49.761115 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:49.761095    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12993h42m17.225786778s"
Jul 02 13:13:50.795769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:50.795406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal" event={"ID":"e3b8679aa0e9fabe089c7dff63237f97","Type":"ContainerStarted","Data":"6afffebf26be74fb65fe0cdadbf6c148ac6cf35f50e785d24691e6399a1b8bc2"}
Jul 02 13:13:50.805793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:50.805755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal" event={"ID":"eb7a4c8ad16d6f669b998210c1cb6767","Type":"ContainerStarted","Data":"705055e4fb1e6aa592722dc1edd4a8ca0c56947202f97a22d91d02b8caa8f90a"}
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:51.437504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:51.437589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:51.437617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.437790    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.437809    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.437822    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.437882    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:55.437864413 +0000 UTC m=+10.168902523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.438306    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.438360    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:13:55.438344462 +0000 UTC m=+10.169382576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.438419    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:51.438492 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.438449    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:13:55.438438151 +0000 UTC m=+10.169476264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:51.755758 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:51.755674    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:51.755928 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.755830    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:51.756250 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:51.756229    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:51.756357 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.756328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:13:51.756572 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:51.756555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:51.756664 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:51.756644    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:52.813972 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:52.813109    2564 generic.go:356] "Generic (PLEG): container finished" podID="e3b8679aa0e9fabe089c7dff63237f97" containerID="6afffebf26be74fb65fe0cdadbf6c148ac6cf35f50e785d24691e6399a1b8bc2" exitCode=0
Jul 02 13:13:52.813972 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:52.813175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal" event={"ID":"e3b8679aa0e9fabe089c7dff63237f97","Type":"ContainerDied","Data":"6afffebf26be74fb65fe0cdadbf6c148ac6cf35f50e785d24691e6399a1b8bc2"}
Jul 02 13:13:52.845568 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:52.844688    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal" podStartSLOduration=5.844664814 podStartE2EDuration="5.844664814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:13:50.836311584 +0000 UTC m=+5.567349735" watchObservedRunningTime="2026-07-02 13:13:52.844664814 +0000 UTC m=+7.575702945"
Jul 02 13:13:53.755064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:53.755024    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:53.755245 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:53.755029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:53.755245 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:53.755165    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:53.755353 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:53.755251    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:53.755353 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:53.755300    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:53.755470 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:53.755371    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:55.475577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:55.475645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:55.475676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.475839    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.475861    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.475874    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.475935    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:03.475915125 +0000 UTC m=+18.206953235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.476012    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.476046    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:03.476034866 +0000 UTC m=+18.207072980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.476103    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:55.476481 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.476136    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:03.47612421 +0000 UTC m=+18.207162319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:13:55.756425 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:55.755716    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:55.756425 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.755826    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:13:55.756425 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:55.756224    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:55.756425 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.756323    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:55.756979 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:55.756833    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:55.756979 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:55.756944    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:57.755322 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:57.754418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:57.755322 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:57.754418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:57.755322 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:57.754648    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:57.755322 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:57.754741    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:13:57.755322 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:57.754826    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:57.755322 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:57.754852    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:59.754603 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:59.754567    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:13:59.755067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:59.754569    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:13:59.755067 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:59.754697    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:13:59.755067 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:59.754783    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:13:59.755067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:13:59.754574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:13:59.755067 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:13:59.754878    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:01.754749 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:01.754714    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:01.755174 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:01.754836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:01.755174 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:01.754849    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:01.755174 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:01.754955    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:01.755174 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:01.755000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:01.755174 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:01.755072    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:03.538342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:03.538257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:03.538342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:03.538308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:03.538342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:03.538332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538409    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538471    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538476    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538492    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538504    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538476    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:19.53846059 +0000 UTC m=+34.269498701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538565    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:19.538522073 +0000 UTC m=+34.269560185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:03.538900 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.538597    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:19.5385844 +0000 UTC m=+34.269622522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:03.754485 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:03.754426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:03.754485 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:03.754469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:03.754751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:03.754425    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:03.754751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.754585    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:03.754751 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.754735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:03.754874 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:03.754806    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:05.754822 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:05.754778    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:05.755292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:05.754923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:05.755292 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:05.754926    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:05.755292 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:05.755033    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:05.755292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:05.755046    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:05.755292 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:05.755128    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:07.754881 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.754608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:07.755490 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.754604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:07.755490 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:07.754947    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:07.755490 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.754604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:07.755490 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:07.755011    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:07.755490 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:07.755055    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:07.847223 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.847060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8qzd5" event={"ID":"1554f3e5-e6ba-4f9e-8211-c43c57869d19","Type":"ContainerStarted","Data":"a0babfd58d100b96c7333e8fe7da684f5a90b2a81b9497acfd9901ccbe473ef3"}
Jul 02 13:14:07.851167 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.851102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wddlr" event={"ID":"f8369f04-06c1-484f-b940-555722659bf6","Type":"ContainerStarted","Data":"0e20b701a89824ed1c0453a1b6c8749893ea4a29282cf5bc0d944731df6c492b"}
Jul 02 13:14:07.852827 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.852795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p55mb" event={"ID":"ce7f1930-b1ee-41b9-a3a9-076991fdd836","Type":"ContainerStarted","Data":"8746b77e66d06f10e5dcc5b7da73ee13ef372202077c73ba3c8f5eaec7bbc970"}
Jul 02 13:14:07.854857 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.854833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal" event={"ID":"e3b8679aa0e9fabe089c7dff63237f97","Type":"ContainerStarted","Data":"28c8c1710266f731961c81b8d2c04395faeb526a0a627ced388af71b45f71246"}
Jul 02 13:14:07.856218 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.856197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-224qj" event={"ID":"bb3278bb-67d1-4e15-a8d6-30b5b204ddcb","Type":"ContainerStarted","Data":"df3e54e6a9dca8cd4fc9c7cd6d6b59d20054f24740ee3b69035a54060b3dba94"}
Jul 02 13:14:07.857560 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.857541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-klrnt" event={"ID":"e7133a6f-870e-4461-9c40-37e238ac59a2","Type":"ContainerStarted","Data":"3909538449bf41613deb8a250b112942c54a562b9a442b5681d299fe082659ab"}
Jul 02 13:14:07.860852 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.860836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:14:07.861133 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.861113    2564 generic.go:356] "Generic (PLEG): container finished" podID="e3d20200-1c44-48fd-80ad-345e9bc3d8b5" containerID="d22e7b73904594343a2ce5f40bcd4145818a4e26f06c576338cdd1ad8173a79e" exitCode=1
Jul 02 13:14:07.861212 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.861183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"73e14435aeb942faa1df341113e9168f66e8a0ad0f12e0f29602f764208f255e"}
Jul 02 13:14:07.861251 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.861239    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerDied","Data":"d22e7b73904594343a2ce5f40bcd4145818a4e26f06c576338cdd1ad8173a79e"}
Jul 02 13:14:07.861326 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.861314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"b9dd21d01a7db9fd448ba5beb99652844ec8309e4b4cf930e227b3283d926c40"}
Jul 02 13:14:07.862431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.862406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2" event={"ID":"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59","Type":"ContainerStarted","Data":"7bdd107b854513b738ce26b75fa68631362b69c8af2a92a27fbdc64b75f69acf"}
Jul 02 13:14:07.863718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.863697    2564 generic.go:356] "Generic (PLEG): container finished" podID="908c8eb7-5b2b-49bd-8952-b68e85db9c4c" containerID="1e1809e0681180e8054eadac343f215916e8728266ca93cc1a16b96f76be6ae9" exitCode=0
Jul 02 13:14:07.863790 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.863729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerDied","Data":"1e1809e0681180e8054eadac343f215916e8728266ca93cc1a16b96f76be6ae9"}
Jul 02 13:14:07.872181 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.872146    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8qzd5" podStartSLOduration=2.993687808 podStartE2EDuration="21.872135379s" totalImagesPullingTime="18.878447571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:07.871088686 +0000 UTC m=+22.602126837" watchObservedRunningTime="2026-07-02 13:14:07.872135379 +0000 UTC m=+22.603173509"
Jul 02 13:14:07.895090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.895052    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wddlr" podStartSLOduration=2.688286248 podStartE2EDuration="21.895037805s" totalImagesPullingTime="19.206751557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:07.894910147 +0000 UTC m=+22.625948299" watchObservedRunningTime="2026-07-02 13:14:07.895037805 +0000 UTC m=+22.626075934"
Jul 02 13:14:07.943304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.943260    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-224qj" podStartSLOduration=2.721720784 podStartE2EDuration="21.94324299s" totalImagesPullingTime="19.221522206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:07.922122014 +0000 UTC m=+22.653160165" watchObservedRunningTime="2026-07-02 13:14:07.94324299 +0000 UTC m=+22.674281130"
Jul 02 13:14:07.993013 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:07.992968    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-klrnt" podStartSLOduration=2.785397503 podStartE2EDuration="21.992950424s" totalImagesPullingTime="19.207552921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:07.943105745 +0000 UTC m=+22.674143895" watchObservedRunningTime="2026-07-02 13:14:07.992950424 +0000 UTC m=+22.723988555"
Jul 02 13:14:08.020005 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.019962    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-135.ec2.internal" podStartSLOduration=21.019946585 podStartE2EDuration="21.019946585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:08.019908734 +0000 UTC m=+22.750946883" watchObservedRunningTime="2026-07-02 13:14:08.019946585 +0000 UTC m=+22.750984715"
Jul 02 13:14:08.868926 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.868899    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:14:08.869581 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.869283    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"c54e2b17fab22023bb8464218b91fcdc76a1a3427001c5a4be70c6bde5bae916"}
Jul 02 13:14:08.869581 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.869346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"5802096d0a0504f5f6b1f6ae1b86b320761337de6b8b98b963bd8469cdae1e1c"}
Jul 02 13:14:08.869581 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.869379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"834bd013128243e88cb4ad85dc835ec75b66842afb3a532b20a37e7f7a0201ef"}
Jul 02 13:14:08.870689 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.870658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-588qg" event={"ID":"189b2773-1cfd-4c74-89a5-7387b4f5adca","Type":"ContainerStarted","Data":"6d40bfc1c92ac143de6634cc7831f9a6a63a2ba8fe2b51f55c89ab2f54f10961"}
Jul 02 13:14:08.888677 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.888639    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-588qg" podStartSLOduration=7.827731654 podStartE2EDuration="22.888626825s" totalImagesPullingTime="15.060895171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:08.888548775 +0000 UTC m=+23.619586917" watchObservedRunningTime="2026-07-02 13:14:08.888626825 +0000 UTC m=+23.619664966"
Jul 02 13:14:08.889125 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:08.889103    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p55mb" podStartSLOduration=3.665873896 podStartE2EDuration="22.889096735s" totalImagesPullingTime="19.223222839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:08.10079462 +0000 UTC m=+22.831832771" watchObservedRunningTime="2026-07-02 13:14:08.889096735 +0000 UTC m=+23.620134877"
Jul 02 13:14:09.746851 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.746825    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:14:09.754953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.754918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:09.755083 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.754904    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:14:09.746844837Z","UUID":"4322d011-1a27-4d17-aff9-f7568dc44cf4","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:14:09.755083 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.754919    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:09.755083 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.755032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:09.755083 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:09.755028    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:09.755286 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:09.755122    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:09.755286 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:09.755209    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:09.756656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.756566    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:14:09.756656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.756606    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:14:09.874824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:09.874733    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2" event={"ID":"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59","Type":"ContainerStarted","Data":"e310b40ba970492a5579747a1b2970bd26649bbbcc455ef8cae48baecce941db"}
Jul 02 13:14:10.881399 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:10.881366    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:14:10.881870 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:10.881728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"5c2d39c5093b2ae6960130919d842649ec170119de150074deb36d85a3327233"}
Jul 02 13:14:11.127085 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:11.127054    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:14:11.754306 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:11.754061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:11.754503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:11.754069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:11.754503 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:11.754408    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:11.754503 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:11.754488    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:11.754677 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:11.754110    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:11.754677 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:11.754605    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:12.360832 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.360795    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:14:12.361439 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.361421    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:14:12.890054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.890027    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:14:12.890419 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.890393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"841212eba0eaf244b0ccb86ec61fc03fef6e87d3d03061ffad82cf7e0ca5c150"}
Jul 02 13:14:12.890642 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.890626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:14:12.890876 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.890857    2564 scope.go:117] "RemoveContainer" containerID="d22e7b73904594343a2ce5f40bcd4145818a4e26f06c576338cdd1ad8173a79e"
Jul 02 13:14:12.892679 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.892658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2" event={"ID":"e0975e29-ddef-4a4b-ac06-d29c0f1cbb59","Type":"ContainerStarted","Data":"11a81b222432c8c1da7925dabc5108bc4a22f52f1b0a71560b133def8d5ea8a2"}
Jul 02 13:14:12.894584 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.894559    2564 generic.go:356] "Generic (PLEG): container finished" podID="908c8eb7-5b2b-49bd-8952-b68e85db9c4c" containerID="d10e31087db780c776d020ae73368157fa49c95235d4d3afcdbaad6912781401" exitCode=0
Jul 02 13:14:12.894678 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.894617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerDied","Data":"d10e31087db780c776d020ae73368157fa49c95235d4d3afcdbaad6912781401"}
Jul 02 13:14:12.895408 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.895337    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wddlr"
Jul 02 13:14:12.908926 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.908908    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:14:12.993039 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:12.992999    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fzlm2" podStartSLOduration=2.8628412069999998 podStartE2EDuration="26.992981952s" totalImagesPullingTime="24.130140745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:12.954215139 +0000 UTC m=+27.685253285" watchObservedRunningTime="2026-07-02 13:14:12.992981952 +0000 UTC m=+27.724020296"
Jul 02 13:14:13.754733 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.754703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:13.755472 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.754737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:13.755472 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:13.754833    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:13.755472 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:13.754894    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:13.755472 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.754907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:13.755472 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:13.754999    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:13.824013 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.823980    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:14:13.904450 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.904427    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:14:13.904920 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.904883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" event={"ID":"e3d20200-1c44-48fd-80ad-345e9bc3d8b5","Type":"ContainerStarted","Data":"681eab7a7541944338d6714e481b4e2c84adacdc9f425e15ecf808300347ca09"}
Jul 02 13:14:13.905281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.905256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:14:13.922987 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:13.922962    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:14:14.053583 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:14.053513    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87" podStartSLOduration=8.575314779 podStartE2EDuration="28.053498077s" totalImagesPullingTime="19.478183298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:14.006428063 +0000 UTC m=+28.737466212" watchObservedRunningTime="2026-07-02 13:14:14.053498077 +0000 UTC m=+28.784536219"
Jul 02 13:14:14.909845 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:14.909812    2564 generic.go:356] "Generic (PLEG): container finished" podID="908c8eb7-5b2b-49bd-8952-b68e85db9c4c" containerID="6403eceeb70313d61d43f8f2266597be9d2a98901c32896bb0f8b53064f4627a" exitCode=0
Jul 02 13:14:14.910415 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:14.909847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerDied","Data":"6403eceeb70313d61d43f8f2266597be9d2a98901c32896bb0f8b53064f4627a"}
Jul 02 13:14:15.173507 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:15.173435    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q86fn"]
Jul 02 13:14:15.173661 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:15.173600    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:15.173713 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:15.173685    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:15.175736 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:15.175712    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8cn72"]
Jul 02 13:14:15.175891 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:15.175797    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:15.175891 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:15.175873    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:15.179597 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:15.179575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kkm78"]
Jul 02 13:14:15.179692 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:15.179666    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:15.179759 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:15.179737    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:16.754454 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:16.754414    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:16.754454 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:16.754452    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:16.754991 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:16.754520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:16.754991 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:16.754558    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:16.754991 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:16.754642    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:16.754991 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:16.754730    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:16.916928 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:16.916665    2564 generic.go:356] "Generic (PLEG): container finished" podID="908c8eb7-5b2b-49bd-8952-b68e85db9c4c" containerID="40f4bac93cc6decac03bd9dc4258bb15119edf4401432dbd8676f9be54a68d01" exitCode=0
Jul 02 13:14:16.917087 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:16.916735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerDied","Data":"40f4bac93cc6decac03bd9dc4258bb15119edf4401432dbd8676f9be54a68d01"}
Jul 02 13:14:18.754595 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:18.754559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:18.755039 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:18.754560    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:18.755039 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:18.754698    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q86fn" podUID="f11610b8-2546-4823-88c3-8d6564b1aac8"
Jul 02 13:14:18.755039 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:18.754560    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:18.755039 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:18.754800    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kkm78" podUID="03505741-d6c7-4043-b2b0-28878d7c2a5d"
Jul 02 13:14:18.755039 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:18.754851    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cn72" podUID="d1d5800b-f421-4fab-b1a3-2d01fce10d58"
Jul 02 13:14:19.561608 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:19.561577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:19.561608 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:19.561621    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:19.561824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:19.561640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:19.561824 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561740    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:19.561824 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561739    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:19.561824 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561805    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.561788905 +0000 UTC m=+66.292827013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:19.561965 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561740    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:19.561965 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561873    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret podName:03505741-d6c7-4043-b2b0-28878d7c2a5d nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.561861396 +0000 UTC m=+66.292899503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret") pod "global-pull-secret-syncer-kkm78" (UID: "03505741-d6c7-4043-b2b0-28878d7c2a5d") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:19.561965 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561752    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:19.561965 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561890    2564 projected.go:196] Error preparing data for projected volume kube-api-access-kgmzr for pod openshift-network-diagnostics/network-check-target-8cn72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:19.561965 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:19.561911    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr podName:d1d5800b-f421-4fab-b1a3-2d01fce10d58 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.561904902 +0000 UTC m=+66.292943009 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgmzr" (UniqueName: "kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr") pod "network-check-target-8cn72" (UID: "d1d5800b-f421-4fab-b1a3-2d01fce10d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:20.042330 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.042293    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-135.ec2.internal" event="NodeReady"
Jul 02 13:14:20.042977 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.042464    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:14:20.120564 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.120460    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dbc699b69-tkgvm"]
Jul 02 13:14:20.126021 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.126001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.132260 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.132157    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 13:14:20.132260 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.132210    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 13:14:20.132473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.132371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 13:14:20.132720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.132696    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7kgk\""
Jul 02 13:14:20.148493 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.148464    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 13:14:20.163559 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.163520    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wj9zq"]
Jul 02 13:14:20.167879 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.167856    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vnxg6"]
Jul 02 13:14:20.168016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.167999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.172057 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.172035    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.173590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.173504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:14:20.173684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.173622    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:14:20.173684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.173668    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5pfr\""
Jul 02 13:14:20.177084 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.177062    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:14:20.177194 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.177114    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:14:20.190387 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.190369    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:14:20.190495 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.190459    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fkpd6\""
Jul 02 13:14:20.200751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.200731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wj9zq"]
Jul 02 13:14:20.211271 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.211244    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnxg6"]
Jul 02 13:14:20.259271 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.259239    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbc699b69-tkgvm"]
Jul 02 13:14:20.266492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5gd\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-kube-api-access-8z5gd\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.266652 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.266652 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.266652 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-trusted-ca\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.266774 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-installation-pull-secrets\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.266774 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-bound-sa-token\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.266864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbm6\" (UniqueName: \"kubernetes.io/projected/a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4-kube-api-access-llbm6\") pod \"ingress-canary-vnxg6\" (UID: \"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4\") " pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.266864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10094788-5376-4e89-b777-f4a84aee36e7-tmp-dir\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.266935 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266889    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-ca-trust-extracted\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.266935 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66znk\" (UniqueName: \"kubernetes.io/projected/10094788-5376-4e89-b777-f4a84aee36e7-kube-api-access-66znk\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.267025 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-image-registry-private-configuration\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.267025 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.266975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-certificates\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.267100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.267049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4-cert\") pod \"ingress-canary-vnxg6\" (UID: \"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4\") " pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.267100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.267082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10094788-5376-4e89-b777-f4a84aee36e7-config-volume\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.367724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66znk\" (UniqueName: \"kubernetes.io/projected/10094788-5376-4e89-b777-f4a84aee36e7-kube-api-access-66znk\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.367724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-image-registry-private-configuration\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-certificates\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4-cert\") pod \"ingress-canary-vnxg6\" (UID: \"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4\") " pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10094788-5376-4e89-b777-f4a84aee36e7-config-volume\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5gd\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-kube-api-access-8z5gd\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-trusted-ca\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-installation-pull-secrets\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.367966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-bound-sa-token\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.368351 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.367981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbm6\" (UniqueName: \"kubernetes.io/projected/a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4-kube-api-access-llbm6\") pod \"ingress-canary-vnxg6\" (UID: \"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4\") " pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.368351 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.368019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10094788-5376-4e89-b777-f4a84aee36e7-tmp-dir\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.368351 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.368055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-ca-trust-extracted\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.368507 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.368475    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:20.368649 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.368566    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:20.868545518 +0000 UTC m=+35.599583641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:20.368649 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.368636    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:20.368649 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.368649    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:20.368819 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.368686    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:20.868673037 +0000 UTC m=+35.599711150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:20.368983 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.368948    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-certificates\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.369068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.368476    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-ca-trust-extracted\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.369068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.369000    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10094788-5376-4e89-b777-f4a84aee36e7-tmp-dir\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.369301 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.369277    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10094788-5376-4e89-b777-f4a84aee36e7-config-volume\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.369639 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.369614    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-trusted-ca\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.373314 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.373256    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-image-registry-private-configuration\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.373314 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.373277    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-installation-pull-secrets\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.373314 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.373262    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4-cert\") pod \"ingress-canary-vnxg6\" (UID: \"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4\") " pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.390122 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.390089    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-bound-sa-token\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.390814 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.390789    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66znk\" (UniqueName: \"kubernetes.io/projected/10094788-5376-4e89-b777-f4a84aee36e7-kube-api-access-66znk\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.392631 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.392370    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbm6\" (UniqueName: \"kubernetes.io/projected/a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4-kube-api-access-llbm6\") pod \"ingress-canary-vnxg6\" (UID: \"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4\") " pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.392631 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.392616    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5gd\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-kube-api-access-8z5gd\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.484958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.484916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnxg6"
Jul 02 13:14:20.754975 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.754938    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:20.755170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.755064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:20.755380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.755330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:20.762276 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.762253    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:14:20.762467 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.762390    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:14:20.762732 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.762713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zjxr6\""
Jul 02 13:14:20.762856 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.762713    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:14:20.762856 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.762763    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:14:20.763640 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.763614    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-drw6p\""
Jul 02 13:14:20.872420 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.872381    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:20.872637 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:20.872433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:20.872637 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.872587    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:20.872772 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.872646    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:20.872772 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.872662    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:20.872772 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.872671    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:21.872649682 +0000 UTC m=+36.603687790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:20.872772 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:20.872697    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:21.872684958 +0000 UTC m=+36.603723095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:21.194863 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:21.194838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnxg6"]
Jul 02 13:14:21.198077 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:14:21.198045    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04ed4c6_8a71_41a5_b4d6_ab56ca5a68c4.slice/crio-47514582900eff25d783ba858369834f65497e223c2ef6fa96e067502ab990b4 WatchSource:0}: Error finding container 47514582900eff25d783ba858369834f65497e223c2ef6fa96e067502ab990b4: Status 404 returned error can't find the container with id 47514582900eff25d783ba858369834f65497e223c2ef6fa96e067502ab990b4
Jul 02 13:14:21.880196 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:21.880151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:21.880405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:21.880207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:21.880405 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:21.880326    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:21.880405 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:21.880351    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:21.880405 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:21.880408    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:23.880386882 +0000 UTC m=+38.611424990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:21.880665 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:21.880325    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:21.880665 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:21.880483    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:23.88046934 +0000 UTC m=+38.611507453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:21.931036 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:21.930997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnxg6" event={"ID":"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4","Type":"ContainerStarted","Data":"47514582900eff25d783ba858369834f65497e223c2ef6fa96e067502ab990b4"}
Jul 02 13:14:21.934409 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:21.934374    2564 generic.go:356] "Generic (PLEG): container finished" podID="908c8eb7-5b2b-49bd-8952-b68e85db9c4c" containerID="321b3d2847252c489dee55215f89b7ee727c7dd9f14c6c8d07d6c251a99bd421" exitCode=0
Jul 02 13:14:21.934597 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:21.934458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerDied","Data":"321b3d2847252c489dee55215f89b7ee727c7dd9f14c6c8d07d6c251a99bd421"}
Jul 02 13:14:22.940178 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:22.940144    2564 generic.go:356] "Generic (PLEG): container finished" podID="908c8eb7-5b2b-49bd-8952-b68e85db9c4c" containerID="104f9cd9b47084cc88d33a98ed1f2fdcb0a3c8f160b1c9efec34b3c7664c9475" exitCode=0
Jul 02 13:14:22.940685 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:22.940187    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerDied","Data":"104f9cd9b47084cc88d33a98ed1f2fdcb0a3c8f160b1c9efec34b3c7664c9475"}
Jul 02 13:14:23.897601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:23.897558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:23.897768 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:23.897620    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:23.897768 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:23.897641    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:23.897768 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:23.897711    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:27.897690893 +0000 UTC m=+42.628729002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:23.897768 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:23.897754    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:23.897768 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:23.897768    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:23.897933 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:23.897810    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:27.897798228 +0000 UTC m=+42.628836346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:23.946181 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:23.946140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" event={"ID":"908c8eb7-5b2b-49bd-8952-b68e85db9c4c","Type":"ContainerStarted","Data":"cbba7d6b3adebbcde67b022a5d57bdfd6224740b848c58a002ac7097c427f7be"}
Jul 02 13:14:23.978059 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:23.978006    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6bdbq" podStartSLOduration=7.544790046 podStartE2EDuration="37.977987951s" totalImagesPullingTime="30.433197905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:23.975928786 +0000 UTC m=+38.706966937" watchObservedRunningTime="2026-07-02 13:14:23.977987951 +0000 UTC m=+38.709026081"
Jul 02 13:14:24.950890 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:24.950628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnxg6" event={"ID":"a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4","Type":"ContainerStarted","Data":"7057788b31af8108672d1c98849c56b71dd83f98bfa2823f2ca94bb77cf37e31"}
Jul 02 13:14:24.978479 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:24.977857    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vnxg6" podStartSLOduration=1.377311219 podStartE2EDuration="4.977833774s" totalImagesPullingTime="3.600522555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:24.977571777 +0000 UTC m=+39.708609937" watchObservedRunningTime="2026-07-02 13:14:24.977833774 +0000 UTC m=+39.708871909"
Jul 02 13:14:27.929067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:27.929022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:27.929067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:27.929068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:27.929664 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:27.929177    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:27.929664 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:27.929194    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:27.929664 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:27.929206    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:27.929664 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:27.929244    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:35.929227477 +0000 UTC m=+50.660265584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:27.929664 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:27.929259    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:35.929253122 +0000 UTC m=+50.660291229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:35.985137 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:35.985094    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:35.985137 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:35.985135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:35.985678 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:35.985227    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:35.985678 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:35.985229    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:35.985678 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:35.985281    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:35.985678 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:35.985282    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.985265723 +0000 UTC m=+66.716303831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:35.985678 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:35.985332    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.985315992 +0000 UTC m=+66.716354100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:45.931038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:45.931009    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfn87"
Jul 02 13:14:51.595665 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.595624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:51.595665 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.595671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:14:51.596155 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.595690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:51.599592 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.599566    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.599702 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.599683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:14:51.599742 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.599684    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:14:51.606853 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.606824    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 13:14:51.606996 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.606896    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs podName:f11610b8-2546-4823-88c3-8d6564b1aac8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:55.60687876 +0000 UTC m=+130.337916868 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs") pod "network-metrics-daemon-q86fn" (UID: "f11610b8-2546-4823-88c3-8d6564b1aac8") : secret "metrics-daemon-secret" not found
Jul 02 13:14:51.609801 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.609786    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.610881 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.610849    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03505741-d6c7-4043-b2b0-28878d7c2a5d-original-pull-secret\") pod \"global-pull-secret-syncer-kkm78\" (UID: \"03505741-d6c7-4043-b2b0-28878d7c2a5d\") " pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:51.620049 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.620027    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgmzr\" (UniqueName: \"kubernetes.io/projected/d1d5800b-f421-4fab-b1a3-2d01fce10d58-kube-api-access-kgmzr\") pod \"network-check-target-8cn72\" (UID: \"d1d5800b-f421-4fab-b1a3-2d01fce10d58\") " pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:51.666026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.666004    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kkm78"
Jul 02 13:14:51.676771 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.676750    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zjxr6\""
Jul 02 13:14:51.684464 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.684441    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:51.802215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.802188    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kkm78"]
Jul 02 13:14:51.805713 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:14:51.805688    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03505741_d6c7_4043_b2b0_28878d7c2a5d.slice/crio-27a238030cd5c7fb26a4126c67b41b62b1f79db92847d7d170104ea8f47c0b22 WatchSource:0}: Error finding container 27a238030cd5c7fb26a4126c67b41b62b1f79db92847d7d170104ea8f47c0b22: Status 404 returned error can't find the container with id 27a238030cd5c7fb26a4126c67b41b62b1f79db92847d7d170104ea8f47c0b22
Jul 02 13:14:51.816949 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.816926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8cn72"]
Jul 02 13:14:51.819338 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:14:51.819320    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d5800b_f421_4fab_b1a3_2d01fce10d58.slice/crio-a0aedd29864860acbbd3d1aa120bbd9fffa0ef64eb1250e7aa64d9708164c402 WatchSource:0}: Error finding container a0aedd29864860acbbd3d1aa120bbd9fffa0ef64eb1250e7aa64d9708164c402: Status 404 returned error can't find the container with id a0aedd29864860acbbd3d1aa120bbd9fffa0ef64eb1250e7aa64d9708164c402
Jul 02 13:14:51.999038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.999005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:14:51.999038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:51.999039    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:14:51.999252 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.999158    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:14:51.999252 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.999216    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:23.999201791 +0000 UTC m=+98.730239904 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:14:51.999252 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.999160    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:14:51.999374 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.999262    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:14:51.999374 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:51.999303    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:23.99929038 +0000 UTC m=+98.730328494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:14:52.016507 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:52.016475    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8cn72" event={"ID":"d1d5800b-f421-4fab-b1a3-2d01fce10d58","Type":"ContainerStarted","Data":"a0aedd29864860acbbd3d1aa120bbd9fffa0ef64eb1250e7aa64d9708164c402"}
Jul 02 13:14:52.017591 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:52.017566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kkm78" event={"ID":"03505741-d6c7-4043-b2b0-28878d7c2a5d","Type":"ContainerStarted","Data":"27a238030cd5c7fb26a4126c67b41b62b1f79db92847d7d170104ea8f47c0b22"}
Jul 02 13:14:55.029198 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:55.029151    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8cn72" event={"ID":"d1d5800b-f421-4fab-b1a3-2d01fce10d58","Type":"ContainerStarted","Data":"9d1865436324074fb530a9000c57200c2a7f60f2cbe13a6f2c3d32fc5214cbfe"}
Jul 02 13:14:55.029715 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:55.029295    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:14:55.054626 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:55.054577    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8cn72" podStartSLOduration=66.284107574 podStartE2EDuration="1m9.054560426s" totalImagesPullingTime="2.770452852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:55.053954794 +0000 UTC m=+69.784992956" watchObservedRunningTime="2026-07-02 13:14:55.054560426 +0000 UTC m=+69.785598556"
Jul 02 13:14:58.042739 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:58.042698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kkm78" event={"ID":"03505741-d6c7-4043-b2b0-28878d7c2a5d","Type":"ContainerStarted","Data":"abe08b2d90ef92440216b0500ca40751b53a3acb12c2c834ed52ef72c1d3be79"}
Jul 02 13:14:58.067537 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:58.067482    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kkm78" podStartSLOduration=65.99760502 podStartE2EDuration="1m12.067463112s" totalImagesPullingTime="6.069858092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:58.067015385 +0000 UTC m=+72.798053540" watchObservedRunningTime="2026-07-02 13:14:58.067463112 +0000 UTC m=+72.798501242"
Jul 02 13:14:59.193580 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.193548    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77ccfbd465-p22wf"]
Jul 02 13:14:59.196621 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.196604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.207015 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.206995    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 02 13:14:59.207111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.206995    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 02 13:14:59.208965 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.208946    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 02 13:14:59.209012 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.208993    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 02 13:14:59.209210 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.209197    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c25qt\""
Jul 02 13:14:59.209249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.209240    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 13:14:59.209301 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.209287    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 13:14:59.218057 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.218034    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77ccfbd465-p22wf"]
Jul 02 13:14:59.253184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.253153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-stats-auth\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.253309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.253225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qj8\" (UniqueName: \"kubernetes.io/projected/c10eca5a-056c-4119-bb57-85fa2cbe1714-kube-api-access-n9qj8\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.253309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.253254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-default-certificate\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.253309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.253296    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.253450 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.253355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.354676 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.354639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qj8\" (UniqueName: \"kubernetes.io/projected/c10eca5a-056c-4119-bb57-85fa2cbe1714-kube-api-access-n9qj8\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.354676 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.354678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-default-certificate\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.354903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.354737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.354903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.354799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.354903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.354820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-stats-auth\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.355043 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:59.354917    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:14:59.355043 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:59.354972    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:59.854952146 +0000 UTC m=+74.585990276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : configmap references non-existent config key: service-ca.crt
Jul 02 13:14:59.355043 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:59.354997    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:59.85498739 +0000 UTC m=+74.586025503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : secret "router-metrics-certs-default" not found
Jul 02 13:14:59.357182 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.357165    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-default-certificate\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.357244 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.357167    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-stats-auth\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.369320 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.369291    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qj8\" (UniqueName: \"kubernetes.io/projected/c10eca5a-056c-4119-bb57-85fa2cbe1714-kube-api-access-n9qj8\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.858754 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.858715    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.858998 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:14:59.858769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:14:59.858998 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:59.858857    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:14:59.858998 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:59.858875    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:00.858860942 +0000 UTC m=+75.589899070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : configmap references non-existent config key: service-ca.crt
Jul 02 13:14:59.858998 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:14:59.858917    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:00.858902697 +0000 UTC m=+75.589940804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : secret "router-metrics-certs-default" not found
Jul 02 13:15:00.864841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:00.864801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:00.865247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:00.864855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:00.865247 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:00.864955    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:00.865247 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:00.864963    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.864950193 +0000 UTC m=+77.595988300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:00.865247 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:00.865022    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.865005898 +0000 UTC m=+77.596044010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : secret "router-metrics-certs-default" not found
Jul 02 13:15:02.878795 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.878752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:02.879213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.878809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:02.879213 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:02.878911    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.878896615 +0000 UTC m=+81.609934724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:02.879213 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:02.878922    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:02.879213 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:02.878991    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.878971765 +0000 UTC m=+81.610009892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : secret "router-metrics-certs-default" not found
Jul 02 13:15:02.905793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.905767    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pwm2v"]
Jul 02 13:15:02.907950 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.907931    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:02.918136 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.918111    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8kq9n\""
Jul 02 13:15:02.934412 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.934388    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:15:02.934489 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.934422    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:15:02.934617 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.934599    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:15:02.934712 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.934696    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:15:02.947126 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.947106    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pwm2v"]
Jul 02 13:15:02.979438 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.979411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:02.979562 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.979454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0486cd46-5551-4638-a225-4925d0eaf8e4-crio-socket\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:02.979562 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.979490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0486cd46-5551-4638-a225-4925d0eaf8e4-data-volume\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:02.979562 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.979536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:02.979674 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:02.979564    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzc4\" (UniqueName: \"kubernetes.io/projected/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-api-access-7dzc4\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080761 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.080730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.080777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0486cd46-5551-4638-a225-4925d0eaf8e4-crio-socket\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.080802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0486cd46-5551-4638-a225-4925d0eaf8e4-data-volume\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.080819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.080837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzc4\" (UniqueName: \"kubernetes.io/projected/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-api-access-7dzc4\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.080855    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.080866    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0486cd46-5551-4638-a225-4925d0eaf8e4-crio-socket\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.080936 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.080920    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:03.580898875 +0000 UTC m=+78.311936990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : configmap "kube-rbac-proxy" not found
Jul 02 13:15:03.081305 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.080984    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:03.081305 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.081037    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:03.581021883 +0000 UTC m=+78.312060004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:03.081305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.081216    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0486cd46-5551-4638-a225-4925d0eaf8e4-data-volume\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.091687 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.091663    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzc4\" (UniqueName: \"kubernetes.io/projected/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-api-access-7dzc4\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.584041 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.584006    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.584234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:03.584079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:03.584234 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.584153    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:15:03.584234 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.584185    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:03.584234 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.584227    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.584208022 +0000 UTC m=+79.315246141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : configmap "kube-rbac-proxy" not found
Jul 02 13:15:03.584379 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:03.584241    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.584235182 +0000 UTC m=+79.315273290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:04.210506 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:04.210473    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8qzd5_1554f3e5-e6ba-4f9e-8211-c43c57869d19/dns-node-resolver/0.log"
Jul 02 13:15:04.592432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:04.592314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:04.592432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:04.592415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:04.592705 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:04.592456    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:15:04.592705 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:04.592555    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.592513796 +0000 UTC m=+81.323551922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : configmap "kube-rbac-proxy" not found
Jul 02 13:15:04.592705 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:04.592555    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:04.592705 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:04.592595    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.592584828 +0000 UTC m=+81.323622938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:05.410897 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:05.410873    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-klrnt_e7133a6f-870e-4461-9c40-37e238ac59a2/node-ca/0.log"
Jul 02 13:15:06.013119 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:06.013089    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnxg6_a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4/serve-healthcheck-canary/0.log"
Jul 02 13:15:06.613134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:06.613085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:06.613552 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:06.613182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:06.613552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.613238    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:15:06.613552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.613312    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.613297613 +0000 UTC m=+85.344335726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : configmap "kube-rbac-proxy" not found
Jul 02 13:15:06.613552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.613338    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:06.613552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.613404    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.613390989 +0000 UTC m=+85.344429098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:06.915026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:06.914990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:06.915182 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:06.915058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:06.915182 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.915142    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:06.915261 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.915182    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:14.915169383 +0000 UTC m=+89.646207493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:06.915261 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:06.915206    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:14.915192032 +0000 UTC m=+89.646230140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : secret "router-metrics-certs-default" not found
Jul 02 13:15:08.058655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.058620    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"]
Jul 02 13:15:08.062680 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.062662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.066148 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.066118    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 13:15:08.066309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.066221    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l6qk2\""
Jul 02 13:15:08.067549 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.067507    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 13:15:08.067549 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.067542    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:15:08.067763 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.067585    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 13:15:08.085498 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.085477    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"]
Jul 02 13:15:08.125899 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.125786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkd8q\" (UniqueName: \"kubernetes.io/projected/4226ad5c-5308-47fa-bb59-a4f1ad054b23-kube-api-access-qkd8q\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.125899 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.125831    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4226ad5c-5308-47fa-bb59-a4f1ad054b23-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.125899 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.125855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4226ad5c-5308-47fa-bb59-a4f1ad054b23-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.227103 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.227054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkd8q\" (UniqueName: \"kubernetes.io/projected/4226ad5c-5308-47fa-bb59-a4f1ad054b23-kube-api-access-qkd8q\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.227246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.227137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4226ad5c-5308-47fa-bb59-a4f1ad054b23-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.227246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.227170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4226ad5c-5308-47fa-bb59-a4f1ad054b23-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.227741 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.227718    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4226ad5c-5308-47fa-bb59-a4f1ad054b23-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.229492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.229471    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4226ad5c-5308-47fa-bb59-a4f1ad054b23-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.236185 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.236167    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkd8q\" (UniqueName: \"kubernetes.io/projected/4226ad5c-5308-47fa-bb59-a4f1ad054b23-kube-api-access-qkd8q\") pod \"kube-storage-version-migrator-operator-79964ff6f9-dc924\" (UID: \"4226ad5c-5308-47fa-bb59-a4f1ad054b23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.372107 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.372019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"
Jul 02 13:15:08.496245 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:08.496212    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924"]
Jul 02 13:15:08.499753 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:08.499728    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4226ad5c_5308_47fa_bb59_a4f1ad054b23.slice/crio-1a487ef9b479c85f63c765605203d29fdd59bad59de9f4df7c532d72849c9511 WatchSource:0}: Error finding container 1a487ef9b479c85f63c765605203d29fdd59bad59de9f4df7c532d72849c9511: Status 404 returned error can't find the container with id 1a487ef9b479c85f63c765605203d29fdd59bad59de9f4df7c532d72849c9511
Jul 02 13:15:09.008810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.008779    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"]
Jul 02 13:15:09.012827 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.012811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.016922 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.016903    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:15:09.017831 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.017809    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 13:15:09.017929 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.017880    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2nnmh\""
Jul 02 13:15:09.017929 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.017898    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 13:15:09.017929 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.017906    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 13:15:09.028953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.028933    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"]
Jul 02 13:15:09.068779 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.068750    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924" event={"ID":"4226ad5c-5308-47fa-bb59-a4f1ad054b23","Type":"ContainerStarted","Data":"1a487ef9b479c85f63c765605203d29fdd59bad59de9f4df7c532d72849c9511"}
Jul 02 13:15:09.134661 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.134631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-config\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.134791 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.134758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.134834 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.134811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9b9h\" (UniqueName: \"kubernetes.io/projected/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-kube-api-access-g9b9h\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.235309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.235273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9b9h\" (UniqueName: \"kubernetes.io/projected/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-kube-api-access-g9b9h\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.235629 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.235346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-config\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.235629 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.235471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.235965 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.235939    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-config\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.238194 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.238169    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.250216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.250191    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9b9h\" (UniqueName: \"kubernetes.io/projected/34cc1e57-1111-4de7-8d06-9fc1ba5527c1-kube-api-access-g9b9h\") pod \"service-ca-operator-5d7b7dc95c-52vdp\" (UID: \"34cc1e57-1111-4de7-8d06-9fc1ba5527c1\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.322124 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.322049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"
Jul 02 13:15:09.467111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:09.467067    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp"]
Jul 02 13:15:09.470826 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:09.470786    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cc1e57_1111_4de7_8d06_9fc1ba5527c1.slice/crio-be9e052c174908fabe4657130704ac5a4ebbb9fad840bd677a2cca1006d77575 WatchSource:0}: Error finding container be9e052c174908fabe4657130704ac5a4ebbb9fad840bd677a2cca1006d77575: Status 404 returned error can't find the container with id be9e052c174908fabe4657130704ac5a4ebbb9fad840bd677a2cca1006d77575
Jul 02 13:15:10.072819 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:10.072779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp" event={"ID":"34cc1e57-1111-4de7-8d06-9fc1ba5527c1","Type":"ContainerStarted","Data":"be9e052c174908fabe4657130704ac5a4ebbb9fad840bd677a2cca1006d77575"}
Jul 02 13:15:10.650889 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:10.650773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:10.651092 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:10.650924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:10.651092 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:10.650958    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:10.651092 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:10.651049    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:18.651026194 +0000 UTC m=+93.382064307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:10.651092 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:10.651048    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:15:10.651092 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:10.651090    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:18.651080038 +0000 UTC m=+93.382118149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : configmap "kube-rbac-proxy" not found
Jul 02 13:15:11.078087 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:11.077985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924" event={"ID":"4226ad5c-5308-47fa-bb59-a4f1ad054b23","Type":"ContainerStarted","Data":"c05b7374999cbcf6b6f8802925a915f5292d28f9e9aa298c26f47d6ae24e98bc"}
Jul 02 13:15:11.096302 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:11.096242    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924" podStartSLOduration=0.843296428 podStartE2EDuration="3.096219819s" totalImagesPullingTime="2.252923391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.095729635 +0000 UTC m=+85.826767778" watchObservedRunningTime="2026-07-02 13:15:11.096219819 +0000 UTC m=+85.827257945"
Jul 02 13:15:12.499966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.499933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"]
Jul 02 13:15:12.503155 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.503135    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"
Jul 02 13:15:12.506432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.506400    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 13:15:12.506601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.506461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 13:15:12.507317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.507297    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-x6dx8\""
Jul 02 13:15:12.522865 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.522840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"]
Jul 02 13:15:12.570728 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.570698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm58z\" (UniqueName: \"kubernetes.io/projected/1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb-kube-api-access-jm58z\") pod \"migrator-786f9fbc5-xdn7b\" (UID: \"1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"
Jul 02 13:15:12.671672 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.671642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm58z\" (UniqueName: \"kubernetes.io/projected/1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb-kube-api-access-jm58z\") pod \"migrator-786f9fbc5-xdn7b\" (UID: \"1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"
Jul 02 13:15:12.682928 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.682900    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm58z\" (UniqueName: \"kubernetes.io/projected/1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb-kube-api-access-jm58z\") pod \"migrator-786f9fbc5-xdn7b\" (UID: \"1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"
Jul 02 13:15:12.814174 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.814093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"
Jul 02 13:15:12.960344 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:12.960314    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b"]
Jul 02 13:15:12.963541 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:12.963502    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef0b6bc_e587_4ef3_8ac6_30cf3cdacbdb.slice/crio-4a10c44786458a76ec6c224ab8412debe851bfd3504e4bca03202b2022970c94 WatchSource:0}: Error finding container 4a10c44786458a76ec6c224ab8412debe851bfd3504e4bca03202b2022970c94: Status 404 returned error can't find the container with id 4a10c44786458a76ec6c224ab8412debe851bfd3504e4bca03202b2022970c94
Jul 02 13:15:13.087579 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:13.087453    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp" event={"ID":"34cc1e57-1111-4de7-8d06-9fc1ba5527c1","Type":"ContainerStarted","Data":"cb561de2aa5f15e522e7bceb1ea1dfa083a70b12d1070b2c7a51fb7a7dbf70e8"}
Jul 02 13:15:13.088541 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:13.088504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b" event={"ID":"1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb","Type":"ContainerStarted","Data":"4a10c44786458a76ec6c224ab8412debe851bfd3504e4bca03202b2022970c94"}
Jul 02 13:15:14.991793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:14.991706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:14.991793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:14.991749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:14.992198 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:14.991863    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:30.99184922 +0000 UTC m=+105.722887331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:14.992198 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:14.991874    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:14.992198 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:14.991934    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs podName:c10eca5a-056c-4119-bb57-85fa2cbe1714 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:30.991920044 +0000 UTC m=+105.722958153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs") pod "router-default-77ccfbd465-p22wf" (UID: "c10eca5a-056c-4119-bb57-85fa2cbe1714") : secret "router-metrics-certs-default" not found
Jul 02 13:15:15.097072 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:15.097037    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b" event={"ID":"1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb","Type":"ContainerStarted","Data":"3c01bde43c161059d5f31b00a0548f1ac428907b31a69598d46209541536d05b"}
Jul 02 13:15:15.097260 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:15.097145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b" event={"ID":"1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb","Type":"ContainerStarted","Data":"3870e69d62b8fb29f2f85305ac31ee61b9340bfa0b116d6cdc6e1db967f4eac8"}
Jul 02 13:15:15.120410 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:15.120356    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-xdn7b" podStartSLOduration=1.365400329 podStartE2EDuration="3.120341725s" totalImagesPullingTime="1.754941396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:15.118665295 +0000 UTC m=+89.849703439" watchObservedRunningTime="2026-07-02 13:15:15.120341725 +0000 UTC m=+89.851379855"
Jul 02 13:15:15.120965 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:15.120931    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp" podStartSLOduration=4.551690039 podStartE2EDuration="7.120921389s" totalImagesPullingTime="2.56923135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:13.110785158 +0000 UTC m=+87.841823310" watchObservedRunningTime="2026-07-02 13:15:15.120921389 +0000 UTC m=+89.851959610"
Jul 02 13:15:18.719110 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:18.719050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:18.719521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:18.719192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:18.719521 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:18.719204    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:18.719521 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:18.719271    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls podName:0486cd46-5551-4638-a225-4925d0eaf8e4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:34.719251578 +0000 UTC m=+109.450289707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwm2v" (UID: "0486cd46-5551-4638-a225-4925d0eaf8e4") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:15:18.719828 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:18.719808    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0486cd46-5551-4638-a225-4925d0eaf8e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:24.062123 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:24.062089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:15:24.062123 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:24.062126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") pod \"image-registry-7dbc699b69-tkgvm\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") " pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:15:24.062551 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:24.062251    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:24.062551 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:24.062282    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:15:24.062551 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:24.062293    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbc699b69-tkgvm: secret "image-registry-tls" not found
Jul 02 13:15:24.062551 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:24.062327    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls podName:10094788-5376-4e89-b777-f4a84aee36e7 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:28.062310823 +0000 UTC m=+162.793348931 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls") pod "dns-default-wj9zq" (UID: "10094788-5376-4e89-b777-f4a84aee36e7") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:24.062551 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:24.062341    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls podName:b3e619fd-4d48-44d1-97ed-ce98fc7d7657 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:28.062335649 +0000 UTC m=+162.793373757 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls") pod "image-registry-7dbc699b69-tkgvm" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657") : secret "image-registry-tls" not found
Jul 02 13:15:26.034718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:26.034690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8cn72"
Jul 02 13:15:31.020137 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:31.020101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:31.020137 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:31.020138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:31.020776 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:31.020758    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10eca5a-056c-4119-bb57-85fa2cbe1714-service-ca-bundle\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:31.022598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:31.022573    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c10eca5a-056c-4119-bb57-85fa2cbe1714-metrics-certs\") pod \"router-default-77ccfbd465-p22wf\" (UID: \"c10eca5a-056c-4119-bb57-85fa2cbe1714\") " pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:31.305256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:31.305164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:31.446447 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:31.446413    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77ccfbd465-p22wf"]
Jul 02 13:15:31.449140 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:31.449105    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10eca5a_056c_4119_bb57_85fa2cbe1714.slice/crio-d2e20593c0ed72344ccc808f98fb535ea6291593231b0c9f336305b6adf441cd WatchSource:0}: Error finding container d2e20593c0ed72344ccc808f98fb535ea6291593231b0c9f336305b6adf441cd: Status 404 returned error can't find the container with id d2e20593c0ed72344ccc808f98fb535ea6291593231b0c9f336305b6adf441cd
Jul 02 13:15:32.152725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:32.152683    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77ccfbd465-p22wf" event={"ID":"c10eca5a-056c-4119-bb57-85fa2cbe1714","Type":"ContainerStarted","Data":"5cabaed5bb56808c88b305513f0c275bc42aceea7db955a381f50ae8d95a1d42"}
Jul 02 13:15:32.153140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:32.152743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77ccfbd465-p22wf" event={"ID":"c10eca5a-056c-4119-bb57-85fa2cbe1714","Type":"ContainerStarted","Data":"d2e20593c0ed72344ccc808f98fb535ea6291593231b0c9f336305b6adf441cd"}
Jul 02 13:15:32.173789 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:32.173738    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77ccfbd465-p22wf" podStartSLOduration=33.173721265 podStartE2EDuration="33.173721265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:32.172842064 +0000 UTC m=+106.903880193" watchObservedRunningTime="2026-07-02 13:15:32.173721265 +0000 UTC m=+106.904759450"
Jul 02 13:15:32.306147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:32.306111    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:32.308824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:32.308799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:33.155713 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:33.155681    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:33.156983 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:33.156962    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77ccfbd465-p22wf"
Jul 02 13:15:34.751899 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:34.751862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:34.754313 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:34.754290    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0486cd46-5551-4638-a225-4925d0eaf8e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwm2v\" (UID: \"0486cd46-5551-4638-a225-4925d0eaf8e4\") " pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:35.016117 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.016030    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pwm2v"
Jul 02 13:15:35.146274 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.146241    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pwm2v"]
Jul 02 13:15:35.149351 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:35.149327    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0486cd46_5551_4638_a225_4925d0eaf8e4.slice/crio-0304aa99675dbb591d11cc75bac5ecd84ec7236048cee342cdb497b238bcdd23 WatchSource:0}: Error finding container 0304aa99675dbb591d11cc75bac5ecd84ec7236048cee342cdb497b238bcdd23: Status 404 returned error can't find the container with id 0304aa99675dbb591d11cc75bac5ecd84ec7236048cee342cdb497b238bcdd23
Jul 02 13:15:35.161670 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.161639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwm2v" event={"ID":"0486cd46-5551-4638-a225-4925d0eaf8e4","Type":"ContainerStarted","Data":"0304aa99675dbb591d11cc75bac5ecd84ec7236048cee342cdb497b238bcdd23"}
Jul 02 13:15:35.705188 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.705155    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"]
Jul 02 13:15:35.707743 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.707720    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:35.739190 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.739164    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 13:15:35.740583 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.740508    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 13:15:35.740583 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.740521    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 13:15:35.740794 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.740629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dv8pf\""
Jul 02 13:15:35.740794 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.740660    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 02 13:15:35.741616 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.741599    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"]
Jul 02 13:15:35.743566 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.743480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:35.757870 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.757846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f10aaa2-ab79-4739-8fe6-709eebc22231-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-n48r6\" (UID: \"0f10aaa2-ab79-4739-8fe6-709eebc22231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:35.759449 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.759428    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"]
Jul 02 13:15:35.761430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.761413    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.768818 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:35.767930    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-operator-admission-webhook-tls\" is forbidden: User \"system:node:ip-10-0-138-135.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-135.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret"
Jul 02 13:15:35.768818 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:35.768697    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-operator-admission-webhook-dockercfg-gvl75\" is forbidden: User \"system:node:ip-10-0-138-135.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-135.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gvl75\"" type="*v1.Secret"
Jul 02 13:15:35.772839 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.772789    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 13:15:35.772958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.772892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tgzsm\""
Jul 02 13:15:35.779586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.779563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 13:15:35.788324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.788301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"]
Jul 02 13:15:35.846329 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.846298    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"]
Jul 02 13:15:35.858905 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.858876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgd6\" (UniqueName: \"kubernetes.io/projected/d4cbc298-ae8c-470f-b044-533cf1eaaa35-kube-api-access-crgd6\") pod \"managed-serviceaccount-addon-agent-c9c57c778-49bvl\" (UID: \"d4cbc298-ae8c-470f-b044-533cf1eaaa35\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:35.859038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.858909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d869b75a-3b17-4a6a-ba2a-18817bad3c3e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kbdhf\" (UID: \"d869b75a-3b17-4a6a-ba2a-18817bad3c3e\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.859038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.858937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f10aaa2-ab79-4739-8fe6-709eebc22231-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-n48r6\" (UID: \"0f10aaa2-ab79-4739-8fe6-709eebc22231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:35.859038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.859006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d869b75a-3b17-4a6a-ba2a-18817bad3c3e-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-kbdhf\" (UID: \"d869b75a-3b17-4a6a-ba2a-18817bad3c3e\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.859342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.859085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4cbc298-ae8c-470f-b044-533cf1eaaa35-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c9c57c778-49bvl\" (UID: \"d4cbc298-ae8c-470f-b044-533cf1eaaa35\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:35.906325 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.906296    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"]
Jul 02 13:15:35.960068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.959997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4cbc298-ae8c-470f-b044-533cf1eaaa35-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c9c57c778-49bvl\" (UID: \"d4cbc298-ae8c-470f-b044-533cf1eaaa35\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:35.960068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.960031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgd6\" (UniqueName: \"kubernetes.io/projected/d4cbc298-ae8c-470f-b044-533cf1eaaa35-kube-api-access-crgd6\") pod \"managed-serviceaccount-addon-agent-c9c57c778-49bvl\" (UID: \"d4cbc298-ae8c-470f-b044-533cf1eaaa35\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:35.960068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.960047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d869b75a-3b17-4a6a-ba2a-18817bad3c3e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kbdhf\" (UID: \"d869b75a-3b17-4a6a-ba2a-18817bad3c3e\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.960266 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.960092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d869b75a-3b17-4a6a-ba2a-18817bad3c3e-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-kbdhf\" (UID: \"d869b75a-3b17-4a6a-ba2a-18817bad3c3e\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.960698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.960672    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d869b75a-3b17-4a6a-ba2a-18817bad3c3e-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-kbdhf\" (UID: \"d869b75a-3b17-4a6a-ba2a-18817bad3c3e\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.962645 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.962616    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d869b75a-3b17-4a6a-ba2a-18817bad3c3e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kbdhf\" (UID: \"d869b75a-3b17-4a6a-ba2a-18817bad3c3e\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:35.962728 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.962618    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4cbc298-ae8c-470f-b044-533cf1eaaa35-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c9c57c778-49bvl\" (UID: \"d4cbc298-ae8c-470f-b044-533cf1eaaa35\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:35.979421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:35.979397    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgd6\" (UniqueName: \"kubernetes.io/projected/d4cbc298-ae8c-470f-b044-533cf1eaaa35-kube-api-access-crgd6\") pod \"managed-serviceaccount-addon-agent-c9c57c778-49bvl\" (UID: \"d4cbc298-ae8c-470f-b044-533cf1eaaa35\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:36.029885 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:36.029832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"
Jul 02 13:15:36.073941 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:36.073905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"
Jul 02 13:15:36.193514 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:36.193484    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl"]
Jul 02 13:15:36.196508 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:36.196483    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4cbc298_ae8c_470f_b044_533cf1eaaa35.slice/crio-5db56806799d7fa4d12f084dee3bcd3f00fd679436d821e4e91ee99dc5cbf771 WatchSource:0}: Error finding container 5db56806799d7fa4d12f084dee3bcd3f00fd679436d821e4e91ee99dc5cbf771: Status 404 returned error can't find the container with id 5db56806799d7fa4d12f084dee3bcd3f00fd679436d821e4e91ee99dc5cbf771
Jul 02 13:15:36.207961 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:36.207939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf"]
Jul 02 13:15:36.210380 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:36.210317    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd869b75a_3b17_4a6a_ba2a_18817bad3c3e.slice/crio-ce43fca80557f1670bbc059147bcc0b4fef0a0b04fac958e493713bd81f4510c WatchSource:0}: Error finding container ce43fca80557f1670bbc059147bcc0b4fef0a0b04fac958e493713bd81f4510c: Status 404 returned error can't find the container with id ce43fca80557f1670bbc059147bcc0b4fef0a0b04fac958e493713bd81f4510c
Jul 02 13:15:36.815218 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:36.815175    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gvl75\""
Jul 02 13:15:36.859552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:36.859490    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: failed to sync secret cache: timed out waiting for the condition
Jul 02 13:15:36.859780 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:36.859613    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f10aaa2-ab79-4739-8fe6-709eebc22231-tls-certificates podName:0f10aaa2-ab79-4739-8fe6-709eebc22231 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:37.359588557 +0000 UTC m=+112.090626679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0f10aaa2-ab79-4739-8fe6-709eebc22231-tls-certificates") pod "prometheus-operator-admission-webhook-7ddc8784d4-n48r6" (UID: "0f10aaa2-ab79-4739-8fe6-709eebc22231") : failed to sync secret cache: timed out waiting for the condition
Jul 02 13:15:37.013178 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:37.013145    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 02 13:15:37.169923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:37.169883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl" event={"ID":"d4cbc298-ae8c-470f-b044-533cf1eaaa35","Type":"ContainerStarted","Data":"5db56806799d7fa4d12f084dee3bcd3f00fd679436d821e4e91ee99dc5cbf771"}
Jul 02 13:15:37.171176 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:37.171146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf" event={"ID":"d869b75a-3b17-4a6a-ba2a-18817bad3c3e","Type":"ContainerStarted","Data":"ce43fca80557f1670bbc059147bcc0b4fef0a0b04fac958e493713bd81f4510c"}
Jul 02 13:15:37.372715 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:37.372666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f10aaa2-ab79-4739-8fe6-709eebc22231-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-n48r6\" (UID: \"0f10aaa2-ab79-4739-8fe6-709eebc22231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:37.375620 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:37.375593    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f10aaa2-ab79-4739-8fe6-709eebc22231-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-n48r6\" (UID: \"0f10aaa2-ab79-4739-8fe6-709eebc22231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:37.551987 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:37.551900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:41.310194 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:41.310165    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"]
Jul 02 13:15:41.315133 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:41.315080    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f10aaa2_ab79_4739_8fe6_709eebc22231.slice/crio-e5924c5751bc40563a24a263b91b12c4660f06a6ba6c066e9238ed61140025b1 WatchSource:0}: Error finding container e5924c5751bc40563a24a263b91b12c4660f06a6ba6c066e9238ed61140025b1: Status 404 returned error can't find the container with id e5924c5751bc40563a24a263b91b12c4660f06a6ba6c066e9238ed61140025b1
Jul 02 13:15:42.193745 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:42.193707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl" event={"ID":"d4cbc298-ae8c-470f-b044-533cf1eaaa35","Type":"ContainerStarted","Data":"dccb001baa51e4fef76aa4c5191c19b36124a8a16c4ac8e23d59441988899c9b"}
Jul 02 13:15:42.195365 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:42.195293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf" event={"ID":"d869b75a-3b17-4a6a-ba2a-18817bad3c3e","Type":"ContainerStarted","Data":"a649eb6fc5302491f1dda14e038b14529d4b87801c96cbf6893f7b828f38ebb9"}
Jul 02 13:15:42.196927 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:42.196900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwm2v" event={"ID":"0486cd46-5551-4638-a225-4925d0eaf8e4","Type":"ContainerStarted","Data":"9a0f33de6c191cd55a23b655abd4602e0039674e28c699ccd7ff4a0c7cdfa7fa"}
Jul 02 13:15:42.197056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:42.196951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwm2v" event={"ID":"0486cd46-5551-4638-a225-4925d0eaf8e4","Type":"ContainerStarted","Data":"3abba7a1cec2bc6ff91bcf85417ca219fb040aa4dc2eb8ace4e579b93fbdb91e"}
Jul 02 13:15:42.197927 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:42.197901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6" event={"ID":"0f10aaa2-ab79-4739-8fe6-709eebc22231","Type":"ContainerStarted","Data":"e5924c5751bc40563a24a263b91b12c4660f06a6ba6c066e9238ed61140025b1"}
Jul 02 13:15:42.222366 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:42.222315    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c9c57c778-49bvl" podStartSLOduration=2.185335365 podStartE2EDuration="7.22229802s" totalImagesPullingTime="5.036962655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:42.221845498 +0000 UTC m=+116.952883629" watchObservedRunningTime="2026-07-02 13:15:42.22229802 +0000 UTC m=+116.953336155"
Jul 02 13:15:43.204065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:43.204027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6" event={"ID":"0f10aaa2-ab79-4739-8fe6-709eebc22231","Type":"ContainerStarted","Data":"f28938901a535e2b382e0bf602aa231a2de56c1eed0f73834f39e56e7e2c2164"}
Jul 02 13:15:43.204560 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:43.204262    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:43.210051 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:43.210019    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6"
Jul 02 13:15:43.224333 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:43.224281    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-n48r6" podStartSLOduration=7.058108249 podStartE2EDuration="8.224265817s" totalImagesPullingTime="1.166157568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:43.22377954 +0000 UTC m=+117.954817695" watchObservedRunningTime="2026-07-02 13:15:43.224265817 +0000 UTC m=+117.955303950"
Jul 02 13:15:43.224476 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:43.224363    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kbdhf" podStartSLOduration=3.266867825 podStartE2EDuration="8.224357368s" totalImagesPullingTime="4.957489543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:42.245426085 +0000 UTC m=+116.976464244" watchObservedRunningTime="2026-07-02 13:15:43.224357368 +0000 UTC m=+117.955395503"
Jul 02 13:15:45.213707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:45.213606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwm2v" event={"ID":"0486cd46-5551-4638-a225-4925d0eaf8e4","Type":"ContainerStarted","Data":"b2100a7732cb11bab5ed6ee9491dc75a922bbb79f93472eacb3606435a1f1f9b"}
Jul 02 13:15:45.243713 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:45.243662    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pwm2v" podStartSLOduration=33.678743279 podStartE2EDuration="43.243647084s" totalImagesPullingTime="9.564903805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:45.242700333 +0000 UTC m=+119.973738479" watchObservedRunningTime="2026-07-02 13:15:45.243647084 +0000 UTC m=+119.974685214"
Jul 02 13:15:50.383311 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.383278    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vbjmw"]
Jul 02 13:15:50.387856 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.387839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.391000 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.390977    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:15:50.391139 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.391036    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4hzn\""
Jul 02 13:15:50.391139 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.391066    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:15:50.391248 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.391216    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:15:50.392504 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.392486    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:15:50.392578 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.392500    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:15:50.392578 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.392537    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:15:50.488505 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-sys\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48kd\" (UniqueName: \"kubernetes.io/projected/4cd67921-d031-4799-9f13-fa3c69680741-kube-api-access-m48kd\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-wtmp\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cd67921-d031-4799-9f13-fa3c69680741-metrics-client-ca\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488664    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-root\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488684 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-tls\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-textfile\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.488888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.488780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.589825 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.589788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.589825 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.589829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-textfile\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590024 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.589947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-sys\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48kd\" (UniqueName: \"kubernetes.io/projected/4cd67921-d031-4799-9f13-fa3c69680741-kube-api-access-m48kd\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590185 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590161    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-sys\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590232 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590181    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-textfile\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590232 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590200    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-wtmp\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cd67921-d031-4799-9f13-fa3c69680741-metrics-client-ca\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-root\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-tls\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590468 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590321    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-wtmp\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590468 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590357    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4cd67921-d031-4799-9f13-fa3c69680741-root\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590468 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:50.590392    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 13:15:50.590468 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:50.590454    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-tls podName:4cd67921-d031-4799-9f13-fa3c69680741 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:51.090436403 +0000 UTC m=+125.821474517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-tls") pod "node-exporter-vbjmw" (UID: "4cd67921-d031-4799-9f13-fa3c69680741") : secret "node-exporter-tls" not found
Jul 02 13:15:50.590468 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590452    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.590810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.590780    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cd67921-d031-4799-9f13-fa3c69680741-metrics-client-ca\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.592212 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.592192    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:50.614999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:50.614974    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48kd\" (UniqueName: \"kubernetes.io/projected/4cd67921-d031-4799-9f13-fa3c69680741-kube-api-access-m48kd\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:51.095369 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:51.095333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-tls\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:51.097801 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:51.097770    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4cd67921-d031-4799-9f13-fa3c69680741-node-exporter-tls\") pod \"node-exporter-vbjmw\" (UID: \"4cd67921-d031-4799-9f13-fa3c69680741\") " pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:51.296752 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:51.296719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbjmw"
Jul 02 13:15:51.308427 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:51.308402    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd67921_d031_4799_9f13_fa3c69680741.slice/crio-72885ae4c2ebcff00755e1515c3c37a1b8b53967034246468101c1aff472886d WatchSource:0}: Error finding container 72885ae4c2ebcff00755e1515c3c37a1b8b53967034246468101c1aff472886d: Status 404 returned error can't find the container with id 72885ae4c2ebcff00755e1515c3c37a1b8b53967034246468101c1aff472886d
Jul 02 13:15:52.237604 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:52.237564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbjmw" event={"ID":"4cd67921-d031-4799-9f13-fa3c69680741","Type":"ContainerStarted","Data":"72885ae4c2ebcff00755e1515c3c37a1b8b53967034246468101c1aff472886d"}
Jul 02 13:15:53.242632 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:53.242602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbjmw" event={"ID":"4cd67921-d031-4799-9f13-fa3c69680741","Type":"ContainerStarted","Data":"0c51259552918c048bfb00ec806034e902943016a3757f37f507ad518d6a2aed"}
Jul 02 13:15:54.246908 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:54.246872    2564 generic.go:356] "Generic (PLEG): container finished" podID="4cd67921-d031-4799-9f13-fa3c69680741" containerID="0c51259552918c048bfb00ec806034e902943016a3757f37f507ad518d6a2aed" exitCode=0
Jul 02 13:15:54.247306 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:54.246948    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbjmw" event={"ID":"4cd67921-d031-4799-9f13-fa3c69680741","Type":"ContainerDied","Data":"0c51259552918c048bfb00ec806034e902943016a3757f37f507ad518d6a2aed"}
Jul 02 13:15:55.252293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.252249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbjmw" event={"ID":"4cd67921-d031-4799-9f13-fa3c69680741","Type":"ContainerStarted","Data":"874973bc558d0dacbc603e13cc15c6f99549cadd69eb12896110944d61617025"}
Jul 02 13:15:55.252698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.252349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbjmw" event={"ID":"4cd67921-d031-4799-9f13-fa3c69680741","Type":"ContainerStarted","Data":"57b2307626b2b80e095027f1d978799bc70eb762567e3028c3a590ce1b075d4f"}
Jul 02 13:15:55.275442 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.275393    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vbjmw" podStartSLOduration=3.406036613 podStartE2EDuration="5.27537617s" totalImagesPullingTime="1.805229977s" totalInitContainerRuntime="64.10958ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:55.273917991 +0000 UTC m=+130.004956137" watchObservedRunningTime="2026-07-02 13:15:55.27537617 +0000 UTC m=+130.006414312"
Jul 02 13:15:55.637281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.637241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:15:55.639644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.639612    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11610b8-2546-4823-88c3-8d6564b1aac8-metrics-certs\") pod \"network-metrics-daemon-q86fn\" (UID: \"f11610b8-2546-4823-88c3-8d6564b1aac8\") " pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:15:55.885025 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.884994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-drw6p\""
Jul 02 13:15:55.889569 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:55.889494    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q86fn"
Jul 02 13:15:56.042240 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:56.042209    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q86fn"]
Jul 02 13:15:56.045267 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:15:56.045239    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11610b8_2546_4823_88c3_8d6564b1aac8.slice/crio-ab80e433ede3c9da2a69f482437931e715e50a4dc83e4b22ad359af79931c968 WatchSource:0}: Error finding container ab80e433ede3c9da2a69f482437931e715e50a4dc83e4b22ad359af79931c968: Status 404 returned error can't find the container with id ab80e433ede3c9da2a69f482437931e715e50a4dc83e4b22ad359af79931c968
Jul 02 13:15:56.256572 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:56.256473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q86fn" event={"ID":"f11610b8-2546-4823-88c3-8d6564b1aac8","Type":"ContainerStarted","Data":"ab80e433ede3c9da2a69f482437931e715e50a4dc83e4b22ad359af79931c968"}
Jul 02 13:15:58.062255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.062171    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dbc699b69-tkgvm"]
Jul 02 13:15:58.062618 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:15:58.062449    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm" podUID="b3e619fd-4d48-44d1-97ed-ce98fc7d7657"
Jul 02 13:15:58.269014 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.268984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:15:58.269179 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.269015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q86fn" event={"ID":"f11610b8-2546-4823-88c3-8d6564b1aac8","Type":"ContainerStarted","Data":"89f776a49ca13f61098de44fc9edc0843fc233e62c24e405bc3824f1df4d45a1"}
Jul 02 13:15:58.269179 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.269102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q86fn" event={"ID":"f11610b8-2546-4823-88c3-8d6564b1aac8","Type":"ContainerStarted","Data":"91e4639e5a9a9e5eef71e5cf9c4d3cc2ec966751d155eefe7461424d359927c7"}
Jul 02 13:15:58.275228 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.275206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:15:58.287901 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.287858    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q86fn" podStartSLOduration=130.669611785 podStartE2EDuration="2m12.287844982s" totalImagesPullingTime="1.618233197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:58.286308258 +0000 UTC m=+133.017346414" watchObservedRunningTime="2026-07-02 13:15:58.287844982 +0000 UTC m=+133.018883111"
Jul 02 13:15:58.359421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359347    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-trusted-ca\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359401    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-installation-pull-secrets\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359439    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-ca-trust-extracted\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359466    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-image-registry-private-configuration\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-certificates\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359605    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z5gd\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-kube-api-access-8z5gd\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359633    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-bound-sa-token\") pod \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\" (UID: \"b3e619fd-4d48-44d1-97ed-ce98fc7d7657\") "
Jul 02 13:15:58.359953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359760    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:15:58.359953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359801    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:15:58.360065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.359974    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:15:58.360065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.360007    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-trusted-ca\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:58.360065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.360029    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-ca-trust-extracted\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:58.361991 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.361970    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:15:58.362098 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.361994    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:15:58.362147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.362127    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:15:58.362216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.362196    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-kube-api-access-8z5gd" (OuterVolumeSpecName: "kube-api-access-8z5gd") pod "b3e619fd-4d48-44d1-97ed-ce98fc7d7657" (UID: "b3e619fd-4d48-44d1-97ed-ce98fc7d7657"). InnerVolumeSpecName "kube-api-access-8z5gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:15:58.460776 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.460736    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-image-registry-private-configuration\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:58.460776 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.460770    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-certificates\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:58.460776 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.460782    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5gd\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-kube-api-access-8z5gd\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:58.461010 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.460791    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-bound-sa-token\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:58.461010 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:58.460800    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-installation-pull-secrets\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:59.273125 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:59.273093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbc699b69-tkgvm"
Jul 02 13:15:59.308966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:59.308937    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dbc699b69-tkgvm"]
Jul 02 13:15:59.312946 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:59.312921    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dbc699b69-tkgvm"]
Jul 02 13:15:59.468899 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:59.468865    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3e619fd-4d48-44d1-97ed-ce98fc7d7657-registry-tls\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:15:59.760470 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:15:59.760436    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e619fd-4d48-44d1-97ed-ce98fc7d7657" path="/var/lib/kubelet/pods/b3e619fd-4d48-44d1-97ed-ce98fc7d7657/volumes"
Jul 02 13:16:04.895379 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.895342    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599fdf878b-d8zc7"]
Jul 02 13:16:04.899845 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.899828    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:04.903952 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.903930    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:16:04.904089 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904003    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:16:04.904089 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8dn8k\""
Jul 02 13:16:04.904202 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904098    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:16:04.904202 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904145    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:16:04.904202 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904098    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:16:04.904202 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904165    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:16:04.904365 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.904265    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:16:04.921958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:04.921931    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599fdf878b-d8zc7"]
Jul 02 13:16:05.013723 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.013693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-service-ca\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.013898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.013732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-oauth-config\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.013898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.013758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-config\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.013898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.013862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh85m\" (UniqueName: \"kubernetes.io/projected/234b6c8c-9cbd-4eb7-9021-de62e25b881b-kube-api-access-dh85m\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.014019 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.013908    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-oauth-serving-cert\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.014019 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.013934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-serving-cert\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.114823 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.114788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh85m\" (UniqueName: \"kubernetes.io/projected/234b6c8c-9cbd-4eb7-9021-de62e25b881b-kube-api-access-dh85m\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.114836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-oauth-serving-cert\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.114862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-serving-cert\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.114891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-service-ca\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.114913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-oauth-config\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.114933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-config\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115792 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.115764    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-oauth-serving-cert\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.115767    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-config\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.115888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.115807    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-service-ca\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.117348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.117327    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-serving-cert\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.117437 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.117404    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-oauth-config\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.125191 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.125164    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh85m\" (UniqueName: \"kubernetes.io/projected/234b6c8c-9cbd-4eb7-9021-de62e25b881b-kube-api-access-dh85m\") pod \"console-599fdf878b-d8zc7\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") " pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.208966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.208887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:05.346751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:05.346621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599fdf878b-d8zc7"]
Jul 02 13:16:05.349811 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:16:05.349753    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234b6c8c_9cbd_4eb7_9021_de62e25b881b.slice/crio-1df7e570fe5d128dc805cb11192558277c601bcc2c9be81c95a20f2de74443f4 WatchSource:0}: Error finding container 1df7e570fe5d128dc805cb11192558277c601bcc2c9be81c95a20f2de74443f4: Status 404 returned error can't find the container with id 1df7e570fe5d128dc805cb11192558277c601bcc2c9be81c95a20f2de74443f4
Jul 02 13:16:06.299018 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:06.298976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599fdf878b-d8zc7" event={"ID":"234b6c8c-9cbd-4eb7-9021-de62e25b881b","Type":"ContainerStarted","Data":"1df7e570fe5d128dc805cb11192558277c601bcc2c9be81c95a20f2de74443f4"}
Jul 02 13:16:08.242099 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:08.242070    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 13:16:08.443947 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:08.443912    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnxg6_a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4/serve-healthcheck-canary/0.log"
Jul 02 13:16:09.311412 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:09.311371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599fdf878b-d8zc7" event={"ID":"234b6c8c-9cbd-4eb7-9021-de62e25b881b","Type":"ContainerStarted","Data":"cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054"}
Jul 02 13:16:09.345427 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:09.345378    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599fdf878b-d8zc7" podStartSLOduration=1.969532059 podStartE2EDuration="5.345362717s" totalImagesPullingTime="3.375830658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:09.343860902 +0000 UTC m=+144.074899048" watchObservedRunningTime="2026-07-02 13:16:09.345362717 +0000 UTC m=+144.076400858"
Jul 02 13:16:15.209064 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.209023    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:15.209612 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.209105    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:15.213818 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.213793    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:15.228140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.228120    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c6f9fb9c-hpscz"]
Jul 02 13:16:15.400159 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.400126    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:15.400343 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.400227    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.400343 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.400257    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c6f9fb9c-hpscz"]
Jul 02 13:16:15.408223 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.408199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:16:15.502402 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-oauth-serving-cert\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.502402 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-service-ca\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.502402 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-oauth-config\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.502639 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-serving-cert\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.502639 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-console-config\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.502724 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-trusted-ca-bundle\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.502763 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.502736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5br7k\" (UniqueName: \"kubernetes.io/projected/7c50032c-4fc1-4897-a367-4325461bc0b8-kube-api-access-5br7k\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604163 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-trusted-ca-bundle\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604163 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5br7k\" (UniqueName: \"kubernetes.io/projected/7c50032c-4fc1-4897-a367-4325461bc0b8-kube-api-access-5br7k\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604368 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-oauth-serving-cert\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604368 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-service-ca\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-oauth-config\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604483 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-serving-cert\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-console-config\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.604984 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604961    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-oauth-serving-cert\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.605078 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.604998    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-service-ca\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.605134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.605106    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-trusted-ca-bundle\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.605187 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.605163    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-console-config\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.607046 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.607023    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-oauth-config\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.607128 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.607023    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-serving-cert\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.613911 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.613883    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5br7k\" (UniqueName: \"kubernetes.io/projected/7c50032c-4fc1-4897-a367-4325461bc0b8-kube-api-access-5br7k\") pod \"console-74c6f9fb9c-hpscz\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") " pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.710181 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.710149    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:15.901457 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:15.901431    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c6f9fb9c-hpscz"]
Jul 02 13:16:15.903898 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:16:15.903871    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c50032c_4fc1_4897_a367_4325461bc0b8.slice/crio-365607bdb79fbe56b99516d3bb5372091656a8f73e595eb45c72cefdade05e08 WatchSource:0}: Error finding container 365607bdb79fbe56b99516d3bb5372091656a8f73e595eb45c72cefdade05e08: Status 404 returned error can't find the container with id 365607bdb79fbe56b99516d3bb5372091656a8f73e595eb45c72cefdade05e08
Jul 02 13:16:16.337204 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:16.337162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6f9fb9c-hpscz" event={"ID":"7c50032c-4fc1-4897-a367-4325461bc0b8","Type":"ContainerStarted","Data":"591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd"}
Jul 02 13:16:16.337693 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:16.337219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6f9fb9c-hpscz" event={"ID":"7c50032c-4fc1-4897-a367-4325461bc0b8","Type":"ContainerStarted","Data":"365607bdb79fbe56b99516d3bb5372091656a8f73e595eb45c72cefdade05e08"}
Jul 02 13:16:16.356068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:16.356018    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c6f9fb9c-hpscz" podStartSLOduration=1.3560034060000001 podStartE2EDuration="1.356003406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:16.355671817 +0000 UTC m=+151.086709980" watchObservedRunningTime="2026-07-02 13:16:16.356003406 +0000 UTC m=+151.087041535"
Jul 02 13:16:22.361120 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:22.361087    2564 generic.go:356] "Generic (PLEG): container finished" podID="4226ad5c-5308-47fa-bb59-a4f1ad054b23" containerID="c05b7374999cbcf6b6f8802925a915f5292d28f9e9aa298c26f47d6ae24e98bc" exitCode=0
Jul 02 13:16:22.361556 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:22.361162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924" event={"ID":"4226ad5c-5308-47fa-bb59-a4f1ad054b23","Type":"ContainerDied","Data":"c05b7374999cbcf6b6f8802925a915f5292d28f9e9aa298c26f47d6ae24e98bc"}
Jul 02 13:16:22.361556 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:22.361487    2564 scope.go:117] "RemoveContainer" containerID="c05b7374999cbcf6b6f8802925a915f5292d28f9e9aa298c26f47d6ae24e98bc"
Jul 02 13:16:23.179743 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:16:23.179637    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wj9zq" podUID="10094788-5376-4e89-b777-f4a84aee36e7"
Jul 02 13:16:23.366572 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:23.366508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-dc924" event={"ID":"4226ad5c-5308-47fa-bb59-a4f1ad054b23","Type":"ContainerStarted","Data":"4bab94217d4b1138fc6b38d0c65559d8cfb283a5f2546aa83bccc81b5c4de046"}
Jul 02 13:16:23.367937 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:23.367913    2564 generic.go:356] "Generic (PLEG): container finished" podID="34cc1e57-1111-4de7-8d06-9fc1ba5527c1" containerID="cb561de2aa5f15e522e7bceb1ea1dfa083a70b12d1070b2c7a51fb7a7dbf70e8" exitCode=0
Jul 02 13:16:23.368008 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:23.367939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp" event={"ID":"34cc1e57-1111-4de7-8d06-9fc1ba5527c1","Type":"ContainerDied","Data":"cb561de2aa5f15e522e7bceb1ea1dfa083a70b12d1070b2c7a51fb7a7dbf70e8"}
Jul 02 13:16:23.368008 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:23.368002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:16:23.368212 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:23.368199    2564 scope.go:117] "RemoveContainer" containerID="cb561de2aa5f15e522e7bceb1ea1dfa083a70b12d1070b2c7a51fb7a7dbf70e8"
Jul 02 13:16:24.373666 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:24.373630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-52vdp" event={"ID":"34cc1e57-1111-4de7-8d06-9fc1ba5527c1","Type":"ContainerStarted","Data":"6f142c813d043ce8ba23e4127cefa6f3ac8332c5e840dd8930538d3ae6560fec"}
Jul 02 13:16:25.710836 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:25.710798    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:25.711274 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:25.710887    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:25.715607 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:25.715585    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:26.386092 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:26.386065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:16:26.436991 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:26.436951    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599fdf878b-d8zc7"]
Jul 02 13:16:28.117173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:28.117133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:16:28.119630 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:28.119600    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10094788-5376-4e89-b777-f4a84aee36e7-metrics-tls\") pod \"dns-default-wj9zq\" (UID: \"10094788-5376-4e89-b777-f4a84aee36e7\") " pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:16:28.172356 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:28.172327    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5pfr\""
Jul 02 13:16:28.179472 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:28.179453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:16:28.325608 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:28.325577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wj9zq"]
Jul 02 13:16:28.328679 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:16:28.328652    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10094788_5376_4e89_b777_f4a84aee36e7.slice/crio-f99e5d720ce561d9de3f823ef978ace794ab4be71af0bb8ea781cea4a68c8d11 WatchSource:0}: Error finding container f99e5d720ce561d9de3f823ef978ace794ab4be71af0bb8ea781cea4a68c8d11: Status 404 returned error can't find the container with id f99e5d720ce561d9de3f823ef978ace794ab4be71af0bb8ea781cea4a68c8d11
Jul 02 13:16:28.389879 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:28.389846    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj9zq" event={"ID":"10094788-5376-4e89-b777-f4a84aee36e7","Type":"ContainerStarted","Data":"f99e5d720ce561d9de3f823ef978ace794ab4be71af0bb8ea781cea4a68c8d11"}
Jul 02 13:16:30.399620 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:30.399577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj9zq" event={"ID":"10094788-5376-4e89-b777-f4a84aee36e7","Type":"ContainerStarted","Data":"fa2a5f0724e8ef5c8ca7ed83a20a6ab10fbed6840559519b12374b627d064c41"}
Jul 02 13:16:30.400010 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:30.399681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj9zq" event={"ID":"10094788-5376-4e89-b777-f4a84aee36e7","Type":"ContainerStarted","Data":"62529081e45d12c24929373f1284a823fd307ee0e9c16989e87c39d988485430"}
Jul 02 13:16:30.400010 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:30.399706    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:16:30.419898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:30.419846    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wj9zq" podStartSLOduration=128.705305135 podStartE2EDuration="2m10.419830811s" totalImagesPullingTime="1.714525676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:30.418746597 +0000 UTC m=+165.149784748" watchObservedRunningTime="2026-07-02 13:16:30.419830811 +0000 UTC m=+165.150868998"
Jul 02 13:16:40.406614 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:40.406504    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wj9zq"
Jul 02 13:16:51.461309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.461245    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-599fdf878b-d8zc7" podUID="234b6c8c-9cbd-4eb7-9021-de62e25b881b" containerName="console" containerID="cri-o://cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054" gracePeriod=15
Jul 02 13:16:51.713356 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.713289    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:51.799626 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.799598    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-oauth-config\") pod \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") "
Jul 02 13:16:51.799817 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.799665    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-service-ca\") pod \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") "
Jul 02 13:16:51.799817 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.799695    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-oauth-serving-cert\") pod \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") "
Jul 02 13:16:51.799817 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.799722    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh85m\" (UniqueName: \"kubernetes.io/projected/234b6c8c-9cbd-4eb7-9021-de62e25b881b-kube-api-access-dh85m\") pod \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") "
Jul 02 13:16:51.799817 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.799758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-config\") pod \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") "
Jul 02 13:16:51.800016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.799837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-serving-cert\") pod \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\" (UID: \"234b6c8c-9cbd-4eb7-9021-de62e25b881b\") "
Jul 02 13:16:51.800187 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.800159    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-service-ca" (OuterVolumeSpecName: "service-ca") pod "234b6c8c-9cbd-4eb7-9021-de62e25b881b" (UID: "234b6c8c-9cbd-4eb7-9021-de62e25b881b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:51.800187 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.800175    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "234b6c8c-9cbd-4eb7-9021-de62e25b881b" (UID: "234b6c8c-9cbd-4eb7-9021-de62e25b881b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:51.800353 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.800175    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-config" (OuterVolumeSpecName: "console-config") pod "234b6c8c-9cbd-4eb7-9021-de62e25b881b" (UID: "234b6c8c-9cbd-4eb7-9021-de62e25b881b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:51.802022 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.801992    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234b6c8c-9cbd-4eb7-9021-de62e25b881b-kube-api-access-dh85m" (OuterVolumeSpecName: "kube-api-access-dh85m") pod "234b6c8c-9cbd-4eb7-9021-de62e25b881b" (UID: "234b6c8c-9cbd-4eb7-9021-de62e25b881b"). InnerVolumeSpecName "kube-api-access-dh85m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:16:51.802100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.801992    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "234b6c8c-9cbd-4eb7-9021-de62e25b881b" (UID: "234b6c8c-9cbd-4eb7-9021-de62e25b881b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:16:51.802100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.802024    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "234b6c8c-9cbd-4eb7-9021-de62e25b881b" (UID: "234b6c8c-9cbd-4eb7-9021-de62e25b881b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:16:51.900377 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.900330    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-service-ca\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:16:51.900377 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.900372    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-oauth-serving-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:16:51.900377 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.900384    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh85m\" (UniqueName: \"kubernetes.io/projected/234b6c8c-9cbd-4eb7-9021-de62e25b881b-kube-api-access-dh85m\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:16:51.900377 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.900393    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-config\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:16:51.900637 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.900402    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-serving-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:16:51.900637 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:51.900410    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/234b6c8c-9cbd-4eb7-9021-de62e25b881b-console-oauth-config\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:16:52.485093 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.485056    2564 generic.go:356] "Generic (PLEG): container finished" podID="234b6c8c-9cbd-4eb7-9021-de62e25b881b" containerID="cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054" exitCode=0
Jul 02 13:16:52.485475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.485112    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599fdf878b-d8zc7"
Jul 02 13:16:52.485475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.485136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599fdf878b-d8zc7" event={"ID":"234b6c8c-9cbd-4eb7-9021-de62e25b881b","Type":"ContainerDied","Data":"cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054"}
Jul 02 13:16:52.485475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.485207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599fdf878b-d8zc7" event={"ID":"234b6c8c-9cbd-4eb7-9021-de62e25b881b","Type":"ContainerDied","Data":"1df7e570fe5d128dc805cb11192558277c601bcc2c9be81c95a20f2de74443f4"}
Jul 02 13:16:52.485475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.485238    2564 scope.go:117] "RemoveContainer" containerID="cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054"
Jul 02 13:16:52.494757 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.494740    2564 scope.go:117] "RemoveContainer" containerID="cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054"
Jul 02 13:16:52.495017 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:16:52.494997    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054\": container with ID starting with cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054 not found: ID does not exist" containerID="cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054"
Jul 02 13:16:52.495088 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.495024    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054"} err="failed to get container status \"cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054\": rpc error: code = NotFound desc = could not find container \"cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054\": container with ID starting with cbe4b06157cf897dc7c422eba4be4a4f9af999ce1f6de2b5518f3369d05af054 not found: ID does not exist"
Jul 02 13:16:52.508691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.508669    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599fdf878b-d8zc7"]
Jul 02 13:16:52.512966 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:52.512946    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599fdf878b-d8zc7"]
Jul 02 13:16:53.758634 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:16:53.758603    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234b6c8c-9cbd-4eb7-9021-de62e25b881b" path="/var/lib/kubelet/pods/234b6c8c-9cbd-4eb7-9021-de62e25b881b/volumes"
Jul 02 13:17:12.550683 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.550649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d9bbc5b66-xqbfn"]
Jul 02 13:17:12.551146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.551032    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234b6c8c-9cbd-4eb7-9021-de62e25b881b" containerName="console"
Jul 02 13:17:12.551146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.551050    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="234b6c8c-9cbd-4eb7-9021-de62e25b881b" containerName="console"
Jul 02 13:17:12.551146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.551112    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="234b6c8c-9cbd-4eb7-9021-de62e25b881b" containerName="console"
Jul 02 13:17:12.552971 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.552951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.590437 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.590405    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9bbc5b66-xqbfn"]
Jul 02 13:17:12.649515 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-serving-cert\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.649696 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc45\" (UniqueName: \"kubernetes.io/projected/cb5fce27-47f9-4546-ab87-2f03ac191068-kube-api-access-hbc45\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.649696 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649602    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-oauth-config\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.649696 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-oauth-serving-cert\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.649696 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-console-config\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.649852 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-service-ca\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.649852 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.649741    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-trusted-ca-bundle\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750689 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-serving-cert\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750689 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc45\" (UniqueName: \"kubernetes.io/projected/cb5fce27-47f9-4546-ab87-2f03ac191068-kube-api-access-hbc45\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-oauth-config\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-oauth-serving-cert\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750770    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-console-config\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-service-ca\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.750958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.750830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-trusted-ca-bundle\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.751736 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.751704    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-service-ca\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.751850 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.751740    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-console-config\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.751850 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.751704    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-oauth-serving-cert\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.751958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.751939    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-trusted-ca-bundle\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.753440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.753414    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-oauth-config\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.753522 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.753504    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-serving-cert\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.759832 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.759813    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc45\" (UniqueName: \"kubernetes.io/projected/cb5fce27-47f9-4546-ab87-2f03ac191068-kube-api-access-hbc45\") pod \"console-d9bbc5b66-xqbfn\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") " pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:12.862058 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:12.861960    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:13.008095 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:13.008017    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9bbc5b66-xqbfn"]
Jul 02 13:17:13.010828 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:17:13.010800    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5fce27_47f9_4546_ab87_2f03ac191068.slice/crio-845436cb837782e564a98fbb5df29c1e0015c55db060b444c7b10e5e5b0ab2f2 WatchSource:0}: Error finding container 845436cb837782e564a98fbb5df29c1e0015c55db060b444c7b10e5e5b0ab2f2: Status 404 returned error can't find the container with id 845436cb837782e564a98fbb5df29c1e0015c55db060b444c7b10e5e5b0ab2f2
Jul 02 13:17:13.563942 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:13.563898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bbc5b66-xqbfn" event={"ID":"cb5fce27-47f9-4546-ab87-2f03ac191068","Type":"ContainerStarted","Data":"37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2"}
Jul 02 13:17:13.564435 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:13.563963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bbc5b66-xqbfn" event={"ID":"cb5fce27-47f9-4546-ab87-2f03ac191068","Type":"ContainerStarted","Data":"845436cb837782e564a98fbb5df29c1e0015c55db060b444c7b10e5e5b0ab2f2"}
Jul 02 13:17:13.584251 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:13.584194    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d9bbc5b66-xqbfn" podStartSLOduration=1.584176669 podStartE2EDuration="1.584176669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:13.584070483 +0000 UTC m=+208.315108628" watchObservedRunningTime="2026-07-02 13:17:13.584176669 +0000 UTC m=+208.315214799"
Jul 02 13:17:22.862654 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:22.862607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:22.863131 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:22.862675    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:22.867545 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:22.867511    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:23.604250 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:23.604217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:17:23.663241 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:23.663208    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c6f9fb9c-hpscz"]
Jul 02 13:17:48.685011 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:48.684956    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74c6f9fb9c-hpscz" podUID="7c50032c-4fc1-4897-a367-4325461bc0b8" containerName="console" containerID="cri-o://591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd" gracePeriod=15
Jul 02 13:17:48.920126 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:48.920102    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:17:49.024126 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024037    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-oauth-config\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024126 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024080    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-service-ca\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024126 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5br7k\" (UniqueName: \"kubernetes.io/projected/7c50032c-4fc1-4897-a367-4325461bc0b8-kube-api-access-5br7k\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024382 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024155    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-oauth-serving-cert\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024382 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-serving-cert\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024382 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024235    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-console-config\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024382 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024259    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-trusted-ca-bundle\") pod \"7c50032c-4fc1-4897-a367-4325461bc0b8\" (UID: \"7c50032c-4fc1-4897-a367-4325461bc0b8\") "
Jul 02 13:17:49.024760 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024629    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:49.024760 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024723    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:49.024940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024758    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-console-config" (OuterVolumeSpecName: "console-config") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:49.024940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.024801    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:49.026367 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.026340    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c50032c-4fc1-4897-a367-4325461bc0b8-kube-api-access-5br7k" (OuterVolumeSpecName: "kube-api-access-5br7k") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "kube-api-access-5br7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:49.026465 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.026388    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:49.026465 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.026431    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c50032c-4fc1-4897-a367-4325461bc0b8" (UID: "7c50032c-4fc1-4897-a367-4325461bc0b8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:49.125151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125102    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5br7k\" (UniqueName: \"kubernetes.io/projected/7c50032c-4fc1-4897-a367-4325461bc0b8-kube-api-access-5br7k\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.125151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125146    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-oauth-serving-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.125151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125157    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-serving-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.125151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125166    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-console-config\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.125416 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125178    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-trusted-ca-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.125416 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125186    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c50032c-4fc1-4897-a367-4325461bc0b8-console-oauth-config\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.125416 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.125194    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c50032c-4fc1-4897-a367-4325461bc0b8-service-ca\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:17:49.697197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.697162    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c50032c-4fc1-4897-a367-4325461bc0b8" containerID="591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd" exitCode=0
Jul 02 13:17:49.697705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.697207    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6f9fb9c-hpscz"
Jul 02 13:17:49.697705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.697248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6f9fb9c-hpscz" event={"ID":"7c50032c-4fc1-4897-a367-4325461bc0b8","Type":"ContainerDied","Data":"591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd"}
Jul 02 13:17:49.697705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.697312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6f9fb9c-hpscz" event={"ID":"7c50032c-4fc1-4897-a367-4325461bc0b8","Type":"ContainerDied","Data":"365607bdb79fbe56b99516d3bb5372091656a8f73e595eb45c72cefdade05e08"}
Jul 02 13:17:49.697705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.697344    2564 scope.go:117] "RemoveContainer" containerID="591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd"
Jul 02 13:17:49.706407 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.706392    2564 scope.go:117] "RemoveContainer" containerID="591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd"
Jul 02 13:17:49.706708 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:17:49.706691    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd\": container with ID starting with 591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd not found: ID does not exist" containerID="591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd"
Jul 02 13:17:49.706766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.706717    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd"} err="failed to get container status \"591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd\": rpc error: code = NotFound desc = could not find container \"591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd\": container with ID starting with 591b342e7025c39a8d821124c6c744d0901d295ba08b8cb17371364a38130acd not found: ID does not exist"
Jul 02 13:17:49.718905 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.718881    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c6f9fb9c-hpscz"]
Jul 02 13:17:49.721759 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.721736    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c6f9fb9c-hpscz"]
Jul 02 13:17:49.759065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:17:49.759042    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c50032c-4fc1-4897-a367-4325461bc0b8" path="/var/lib/kubelet/pods/7c50032c-4fc1-4897-a367-4325461bc0b8/volumes"
Jul 02 13:18:32.780828 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.780791    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"]
Jul 02 13:18:32.781236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.781030    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c50032c-4fc1-4897-a367-4325461bc0b8" containerName="console"
Jul 02 13:18:32.781236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.781041    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c50032c-4fc1-4897-a367-4325461bc0b8" containerName="console"
Jul 02 13:18:32.781236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.781088    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c50032c-4fc1-4897-a367-4325461bc0b8" containerName="console"
Jul 02 13:18:32.782912 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.782896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:32.785663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.785642    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lktlz\""
Jul 02 13:18:32.785663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.785650    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:18:32.786894 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.786879    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:18:32.792886 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.792868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"]
Jul 02 13:18:32.945846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.945822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:32.946014 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.945863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:32.946014 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:32.945926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvp9\" (UniqueName: \"kubernetes.io/projected/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-kube-api-access-7vvp9\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.047127 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.047070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.047127 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.047108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.047292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.047133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvp9\" (UniqueName: \"kubernetes.io/projected/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-kube-api-access-7vvp9\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.047420 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.047400    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.047514 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.047495    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.055690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.055670    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvp9\" (UniqueName: \"kubernetes.io/projected/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-kube-api-access-7vvp9\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.091609 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.091577    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:33.215666 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.215638    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"]
Jul 02 13:18:33.219564 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:18:33.219519    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb698a57a_2f3d_4a85_97e2_f1fe51dfb901.slice/crio-c0931b5de8a3f942e0269084e8c3e9a13bfb77b7585a1b1d1a117ee44687e9ee WatchSource:0}: Error finding container c0931b5de8a3f942e0269084e8c3e9a13bfb77b7585a1b1d1a117ee44687e9ee: Status 404 returned error can't find the container with id c0931b5de8a3f942e0269084e8c3e9a13bfb77b7585a1b1d1a117ee44687e9ee
Jul 02 13:18:33.855874 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:33.855835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj" event={"ID":"b698a57a-2f3d-4a85-97e2-f1fe51dfb901","Type":"ContainerStarted","Data":"c0931b5de8a3f942e0269084e8c3e9a13bfb77b7585a1b1d1a117ee44687e9ee"}
Jul 02 13:18:39.882838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:39.882799    2564 generic.go:356] "Generic (PLEG): container finished" podID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerID="9c8da5cae99a91ebad8c7b4d48ac0c3effda5ea394d0e765f7a8676214cca0f6" exitCode=0
Jul 02 13:18:39.883299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:39.882883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj" event={"ID":"b698a57a-2f3d-4a85-97e2-f1fe51dfb901","Type":"ContainerDied","Data":"9c8da5cae99a91ebad8c7b4d48ac0c3effda5ea394d0e765f7a8676214cca0f6"}
Jul 02 13:18:43.901409 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:43.901369    2564 generic.go:356] "Generic (PLEG): container finished" podID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerID="908628d494717712a51666fc11f0e5e2aa1ffe3f379ef9db7b18d0c8ed72f2c4" exitCode=0
Jul 02 13:18:43.901883 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:43.901454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj" event={"ID":"b698a57a-2f3d-4a85-97e2-f1fe51dfb901","Type":"ContainerDied","Data":"908628d494717712a51666fc11f0e5e2aa1ffe3f379ef9db7b18d0c8ed72f2c4"}
Jul 02 13:18:45.700428 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:45.700203    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:18:45.704545 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:45.704402    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:18:45.706053 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:45.706035    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:18:53.941706 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:53.941667    2564 generic.go:356] "Generic (PLEG): container finished" podID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerID="a4f0f023280b26589d1aac959c87e1b2861e33d658daa94ebb7d7cca1cb28029" exitCode=0
Jul 02 13:18:53.942116 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:53.941739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj" event={"ID":"b698a57a-2f3d-4a85-97e2-f1fe51dfb901","Type":"ContainerDied","Data":"a4f0f023280b26589d1aac959c87e1b2861e33d658daa94ebb7d7cca1cb28029"}
Jul 02 13:18:55.075283 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.075259    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:55.121554 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.121505    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-bundle\") pod \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") "
Jul 02 13:18:55.121554 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.121560    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-util\") pod \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") "
Jul 02 13:18:55.121750 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.121590    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vvp9\" (UniqueName: \"kubernetes.io/projected/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-kube-api-access-7vvp9\") pod \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\" (UID: \"b698a57a-2f3d-4a85-97e2-f1fe51dfb901\") "
Jul 02 13:18:55.122314 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.122277    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-bundle" (OuterVolumeSpecName: "bundle") pod "b698a57a-2f3d-4a85-97e2-f1fe51dfb901" (UID: "b698a57a-2f3d-4a85-97e2-f1fe51dfb901"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:18:55.123801 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.123779    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-kube-api-access-7vvp9" (OuterVolumeSpecName: "kube-api-access-7vvp9") pod "b698a57a-2f3d-4a85-97e2-f1fe51dfb901" (UID: "b698a57a-2f3d-4a85-97e2-f1fe51dfb901"). InnerVolumeSpecName "kube-api-access-7vvp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:18:55.128733 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.128702    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-util" (OuterVolumeSpecName: "util") pod "b698a57a-2f3d-4a85-97e2-f1fe51dfb901" (UID: "b698a57a-2f3d-4a85-97e2-f1fe51dfb901"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:18:55.222222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.222138    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:18:55.222222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.222169    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:18:55.222222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.222182    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvp9\" (UniqueName: \"kubernetes.io/projected/b698a57a-2f3d-4a85-97e2-f1fe51dfb901-kube-api-access-7vvp9\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:18:55.950977 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.950943    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj"
Jul 02 13:18:55.951186 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.950951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ccqhj" event={"ID":"b698a57a-2f3d-4a85-97e2-f1fe51dfb901","Type":"ContainerDied","Data":"c0931b5de8a3f942e0269084e8c3e9a13bfb77b7585a1b1d1a117ee44687e9ee"}
Jul 02 13:18:55.951186 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:18:55.951099    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0931b5de8a3f942e0269084e8c3e9a13bfb77b7585a1b1d1a117ee44687e9ee"
Jul 02 13:19:00.211330 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211292    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"]
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211567    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="pull"
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211579    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="pull"
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211601    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="util"
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211606    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="util"
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211614    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="extract"
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211620    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="extract"
Jul 02 13:19:00.211708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.211677    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b698a57a-2f3d-4a85-97e2-f1fe51dfb901" containerName="extract"
Jul 02 13:19:00.214825 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.214810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.218389 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.218364    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-czgwg\""
Jul 02 13:19:00.218784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.218751    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 13:19:00.218784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.218775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 13:19:00.218784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.218760    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 13:19:00.232626 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.232601    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"]
Jul 02 13:19:00.259831 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.259796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jcq\" (UniqueName: \"kubernetes.io/projected/2fb33f6c-9827-4d71-8c2d-1496ca48200c-kube-api-access-f2jcq\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-znfpk\" (UID: \"2fb33f6c-9827-4d71-8c2d-1496ca48200c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.259993 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.259908    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2fb33f6c-9827-4d71-8c2d-1496ca48200c-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-znfpk\" (UID: \"2fb33f6c-9827-4d71-8c2d-1496ca48200c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.360810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.360763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2fb33f6c-9827-4d71-8c2d-1496ca48200c-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-znfpk\" (UID: \"2fb33f6c-9827-4d71-8c2d-1496ca48200c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.360985 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.360818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jcq\" (UniqueName: \"kubernetes.io/projected/2fb33f6c-9827-4d71-8c2d-1496ca48200c-kube-api-access-f2jcq\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-znfpk\" (UID: \"2fb33f6c-9827-4d71-8c2d-1496ca48200c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.363164 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.363139    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2fb33f6c-9827-4d71-8c2d-1496ca48200c-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-znfpk\" (UID: \"2fb33f6c-9827-4d71-8c2d-1496ca48200c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.377582 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.377552    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jcq\" (UniqueName: \"kubernetes.io/projected/2fb33f6c-9827-4d71-8c2d-1496ca48200c-kube-api-access-f2jcq\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-znfpk\" (UID: \"2fb33f6c-9827-4d71-8c2d-1496ca48200c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.524377 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.524287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:00.696049 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.696016    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"]
Jul 02 13:19:00.700097 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:19:00.700070    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb33f6c_9827_4d71_8c2d_1496ca48200c.slice/crio-f2dda9c67bb7b162b41a1fa78a557e081ff069ca59799e68ad6b9220abc95eb9 WatchSource:0}: Error finding container f2dda9c67bb7b162b41a1fa78a557e081ff069ca59799e68ad6b9220abc95eb9: Status 404 returned error can't find the container with id f2dda9c67bb7b162b41a1fa78a557e081ff069ca59799e68ad6b9220abc95eb9
Jul 02 13:19:00.701952 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.701932    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:19:00.971272 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:00.971238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk" event={"ID":"2fb33f6c-9827-4d71-8c2d-1496ca48200c","Type":"ContainerStarted","Data":"f2dda9c67bb7b162b41a1fa78a557e081ff069ca59799e68ad6b9220abc95eb9"}
Jul 02 13:19:04.743000 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.742964    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-x8cst"]
Jul 02 13:19:04.746295 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.746276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.749107 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.749082    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 13:19:04.749230 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.749184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 13:19:04.749433 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.749416    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-65stb\""
Jul 02 13:19:04.755309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.755280    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-x8cst"]
Jul 02 13:19:04.797846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.797817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.797846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.797846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-cabundle0\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.798105 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.798078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffcm\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-kube-api-access-6ffcm\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.898493 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.898454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffcm\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-kube-api-access-6ffcm\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.898686 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.898514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.898686 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.898569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-cabundle0\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.898686 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:04.898630    2564 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 02 13:19:04.898686 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:04.898647    2564 secret.go:281] references non-existent secret key: ca.crt
Jul 02 13:19:04.898686 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:04.898656    2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 13:19:04.898686 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:04.898668    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-x8cst: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 13:19:04.898877 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:04.898734    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates podName:ae7c0e4e-0e90-4159-977d-499d6ae9e2ca nodeName:}" failed. No retries permitted until 2026-07-02 13:19:05.398718453 +0000 UTC m=+320.129756564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates") pod "keda-operator-68bff59c-x8cst" (UID: "ae7c0e4e-0e90-4159-977d-499d6ae9e2ca") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 13:19:04.899209 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.899191    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-cabundle0\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.924404 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.924381    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffcm\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-kube-api-access-6ffcm\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:04.988383 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.988342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk" event={"ID":"2fb33f6c-9827-4d71-8c2d-1496ca48200c","Type":"ContainerStarted","Data":"44e27653b20f2d15f86c4fd8830ad7200cc945d9541ce79b8ec0db8de6256e1e"}
Jul 02 13:19:04.988581 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:04.988471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:05.010313 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.010220    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk" podStartSLOduration=1.681610593 podStartE2EDuration="5.010202738s" totalImagesPullingTime="3.328592145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:05.009215906 +0000 UTC m=+319.740254057" watchObservedRunningTime="2026-07-02 13:19:05.010202738 +0000 UTC m=+319.741240867"
Jul 02 13:19:05.403159 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.403129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:05.403370 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:05.403301    2564 secret.go:281] references non-existent secret key: ca.crt
Jul 02 13:19:05.403370 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:05.403324    2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 13:19:05.403370 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:05.403335    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-x8cst: references non-existent secret key: ca.crt
Jul 02 13:19:05.403556 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:05.403398    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates podName:ae7c0e4e-0e90-4159-977d-499d6ae9e2ca nodeName:}" failed. No retries permitted until 2026-07-02 13:19:06.403380331 +0000 UTC m=+321.134418439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates") pod "keda-operator-68bff59c-x8cst" (UID: "ae7c0e4e-0e90-4159-977d-499d6ae9e2ca") : references non-existent secret key: ca.crt
Jul 02 13:19:05.871082 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.870985    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-kb885"]
Jul 02 13:19:05.874405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.874385    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:05.877584 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.877562    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 02 13:19:05.893237 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.893210    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-kb885"]
Jul 02 13:19:05.906154 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.906126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ba1e3bb-731a-4377-95df-51b4e40326fb-certificates\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:05.906268 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.906180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6ba1e3bb-731a-4377-95df-51b4e40326fb-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:05.906268 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:05.906251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkb7w\" (UniqueName: \"kubernetes.io/projected/6ba1e3bb-731a-4377-95df-51b4e40326fb-kube-api-access-wkb7w\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.006846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.006812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ba1e3bb-731a-4377-95df-51b4e40326fb-certificates\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.007026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.006861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6ba1e3bb-731a-4377-95df-51b4e40326fb-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.007026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.006928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkb7w\" (UniqueName: \"kubernetes.io/projected/6ba1e3bb-731a-4377-95df-51b4e40326fb-kube-api-access-wkb7w\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.007329 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.007308    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6ba1e3bb-731a-4377-95df-51b4e40326fb-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.009435 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.009416    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ba1e3bb-731a-4377-95df-51b4e40326fb-certificates\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.016732 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.016712    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkb7w\" (UniqueName: \"kubernetes.io/projected/6ba1e3bb-731a-4377-95df-51b4e40326fb-kube-api-access-wkb7w\") pod \"keda-metrics-apiserver-559f75f947-kb885\" (UID: \"6ba1e3bb-731a-4377-95df-51b4e40326fb\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.109699 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.109669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-f9nnr"]
Jul 02 13:19:06.112739 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.112722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.116116 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.116095    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 13:19:06.125432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.125375    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-f9nnr"]
Jul 02 13:19:06.185270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.185232    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:06.209117 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.209087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxq4\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-kube-api-access-ccxq4\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.209244 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.209123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-certificates\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.309811 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.309777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxq4\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-kube-api-access-ccxq4\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.309811 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.309815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-certificates\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.309998 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:06.309982    2564 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 02 13:19:06.310034 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:06.310009    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-f9nnr: secret "keda-admission-webhooks-certs" not found
Jul 02 13:19:06.310081 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:19:06.310071    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-certificates podName:36f9174d-45ab-4931-ab14-0c3a41d27531 nodeName:}" failed. No retries permitted until 2026-07-02 13:19:06.810053161 +0000 UTC m=+321.541091292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-certificates") pod "keda-admission-5fbd5c4644-f9nnr" (UID: "36f9174d-45ab-4931-ab14-0c3a41d27531") : secret "keda-admission-webhooks-certs" not found
Jul 02 13:19:06.342310 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.342275    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxq4\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-kube-api-access-ccxq4\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.343087 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.343060    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-kb885"]
Jul 02 13:19:06.343566 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:19:06.343520    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba1e3bb_731a_4377_95df_51b4e40326fb.slice/crio-53495f411f234e2e1dd874221704e016a64d8dcbc62d0dc38797e9a35a9745bb WatchSource:0}: Error finding container 53495f411f234e2e1dd874221704e016a64d8dcbc62d0dc38797e9a35a9745bb: Status 404 returned error can't find the container with id 53495f411f234e2e1dd874221704e016a64d8dcbc62d0dc38797e9a35a9745bb
Jul 02 13:19:06.410821 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.410791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:06.413201 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.413183    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae7c0e4e-0e90-4159-977d-499d6ae9e2ca-certificates\") pod \"keda-operator-68bff59c-x8cst\" (UID: \"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca\") " pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:06.556961 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.556920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:06.701715 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.701636    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-x8cst"]
Jul 02 13:19:06.704764 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:19:06.704734    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7c0e4e_0e90_4159_977d_499d6ae9e2ca.slice/crio-84006265f27e669cdd2b919c8ba38b46a38b8c9601cc378a0aa8ea9c528da51a WatchSource:0}: Error finding container 84006265f27e669cdd2b919c8ba38b46a38b8c9601cc378a0aa8ea9c528da51a: Status 404 returned error can't find the container with id 84006265f27e669cdd2b919c8ba38b46a38b8c9601cc378a0aa8ea9c528da51a
Jul 02 13:19:06.814003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.813971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-certificates\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.816428 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.816407    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f9174d-45ab-4931-ab14-0c3a41d27531-certificates\") pod \"keda-admission-5fbd5c4644-f9nnr\" (UID: \"36f9174d-45ab-4931-ab14-0c3a41d27531\") " pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:06.997247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.997149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885" event={"ID":"6ba1e3bb-731a-4377-95df-51b4e40326fb","Type":"ContainerStarted","Data":"53495f411f234e2e1dd874221704e016a64d8dcbc62d0dc38797e9a35a9745bb"}
Jul 02 13:19:06.998340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:06.998319    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-x8cst" event={"ID":"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca","Type":"ContainerStarted","Data":"84006265f27e669cdd2b919c8ba38b46a38b8c9601cc378a0aa8ea9c528da51a"}
Jul 02 13:19:07.022758 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:07.022728    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:07.175413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:07.175364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-f9nnr"]
Jul 02 13:19:07.179379 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:19:07.179350    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9174d_45ab_4931_ab14_0c3a41d27531.slice/crio-7befce115294e551985377d09c4e141fe3ff3e9d8fab7c93e303434366574dea WatchSource:0}: Error finding container 7befce115294e551985377d09c4e141fe3ff3e9d8fab7c93e303434366574dea: Status 404 returned error can't find the container with id 7befce115294e551985377d09c4e141fe3ff3e9d8fab7c93e303434366574dea
Jul 02 13:19:08.004840 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:08.004798    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr" event={"ID":"36f9174d-45ab-4931-ab14-0c3a41d27531","Type":"ContainerStarted","Data":"7befce115294e551985377d09c4e141fe3ff3e9d8fab7c93e303434366574dea"}
Jul 02 13:19:11.021100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.021059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885" event={"ID":"6ba1e3bb-731a-4377-95df-51b4e40326fb","Type":"ContainerStarted","Data":"1431a89d40bad9cd20aef19769e4dbba9f15c43181842aed45e7aa75ecebfd81"}
Jul 02 13:19:11.021595 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.021241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:11.022680 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.022651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr" event={"ID":"36f9174d-45ab-4931-ab14-0c3a41d27531","Type":"ContainerStarted","Data":"c70c0d91c43659a59e5b100f4b9f751d8af93dcb38f3f5d3204f898cc1179687"}
Jul 02 13:19:11.022791 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.022710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:19:11.024177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.024155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-x8cst" event={"ID":"ae7c0e4e-0e90-4159-977d-499d6ae9e2ca","Type":"ContainerStarted","Data":"a07980cc9d2460d09ec8460f892402689f0d0aba7a327241c74ee375ce274b4b"}
Jul 02 13:19:11.024288 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.024276    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:11.046159 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.046115    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885" podStartSLOduration=1.55123838 podStartE2EDuration="6.046100585s" totalImagesPullingTime="4.494862205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:11.04525792 +0000 UTC m=+325.776296072" watchObservedRunningTime="2026-07-02 13:19:11.046100585 +0000 UTC m=+325.777138717"
Jul 02 13:19:11.065015 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.064934    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-x8cst" podStartSLOduration=2.915291462 podStartE2EDuration="7.06492072s" totalImagesPullingTime="4.149629258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:11.063643736 +0000 UTC m=+325.794681887" watchObservedRunningTime="2026-07-02 13:19:11.06492072 +0000 UTC m=+325.795958849"
Jul 02 13:19:11.092730 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:11.092686    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr" podStartSLOduration=1.4372611690000001 podStartE2EDuration="5.092673476s" totalImagesPullingTime="3.655412307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:11.091947054 +0000 UTC m=+325.822985203" watchObservedRunningTime="2026-07-02 13:19:11.092673476 +0000 UTC m=+325.823711667"
Jul 02 13:19:22.033872 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:22.033841    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-kb885"
Jul 02 13:19:25.995310 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:25.995277    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-znfpk"
Jul 02 13:19:32.030974 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:32.030941    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-x8cst"
Jul 02 13:19:32.031401 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:19:32.031256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-f9nnr"
Jul 02 13:20:14.012326 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.012290    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-w6jnj"]
Jul 02 13:20:14.015236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.015217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.017974 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.017953    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:20:14.017974 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.017962    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 13:20:14.018166 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.018079    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:20:14.019375 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.019356    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tw2bd\""
Jul 02 13:20:14.027397 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.027371    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-w6jnj"]
Jul 02 13:20:14.034469 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.034445    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6bbd58757b-88495"]
Jul 02 13:20:14.034797 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.034777    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pl8d\" (UniqueName: \"kubernetes.io/projected/8e30eb35-606a-45ba-912d-7929ad10b3f3-kube-api-access-7pl8d\") pod \"kserve-controller-manager-694f5b7594-w6jnj\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") " pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.034921 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.034905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e30eb35-606a-45ba-912d-7929ad10b3f3-cert\") pod \"kserve-controller-manager-694f5b7594-w6jnj\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") " pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.037559 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.037543    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.040519 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.040497    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 02 13:20:14.040631 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.040606    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xwpbv\""
Jul 02 13:20:14.048886 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.048865    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6bbd58757b-88495"]
Jul 02 13:20:14.135369 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.135339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e30eb35-606a-45ba-912d-7929ad10b3f3-cert\") pod \"kserve-controller-manager-694f5b7594-w6jnj\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") " pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.135558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.135374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pl8d\" (UniqueName: \"kubernetes.io/projected/8e30eb35-606a-45ba-912d-7929ad10b3f3-kube-api-access-7pl8d\") pod \"kserve-controller-manager-694f5b7594-w6jnj\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") " pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.135558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.135403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38758751-9e67-4e38-847f-e0c26baa24dd-cert\") pod \"llmisvc-controller-manager-6bbd58757b-88495\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") " pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.135558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.135436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvd7\" (UniqueName: \"kubernetes.io/projected/38758751-9e67-4e38-847f-e0c26baa24dd-kube-api-access-4rvd7\") pod \"llmisvc-controller-manager-6bbd58757b-88495\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") " pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.137956 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.137929    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e30eb35-606a-45ba-912d-7929ad10b3f3-cert\") pod \"kserve-controller-manager-694f5b7594-w6jnj\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") " pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.151691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.151666    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pl8d\" (UniqueName: \"kubernetes.io/projected/8e30eb35-606a-45ba-912d-7929ad10b3f3-kube-api-access-7pl8d\") pod \"kserve-controller-manager-694f5b7594-w6jnj\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") " pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.236465 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.236429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvd7\" (UniqueName: \"kubernetes.io/projected/38758751-9e67-4e38-847f-e0c26baa24dd-kube-api-access-4rvd7\") pod \"llmisvc-controller-manager-6bbd58757b-88495\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") " pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.236656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.236556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38758751-9e67-4e38-847f-e0c26baa24dd-cert\") pod \"llmisvc-controller-manager-6bbd58757b-88495\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") " pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.238873 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.238849    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38758751-9e67-4e38-847f-e0c26baa24dd-cert\") pod \"llmisvc-controller-manager-6bbd58757b-88495\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") " pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.245810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.245787    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvd7\" (UniqueName: \"kubernetes.io/projected/38758751-9e67-4e38-847f-e0c26baa24dd-kube-api-access-4rvd7\") pod \"llmisvc-controller-manager-6bbd58757b-88495\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") " pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.325147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.325067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:14.346689 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.346644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:14.471338 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.471286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-w6jnj"]
Jul 02 13:20:14.473747 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:20:14.473711    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e30eb35_606a_45ba_912d_7929ad10b3f3.slice/crio-f73e959eb789e07be29377c25fdaadbe86b0ae35e89ebbfd667a0a555fce492a WatchSource:0}: Error finding container f73e959eb789e07be29377c25fdaadbe86b0ae35e89ebbfd667a0a555fce492a: Status 404 returned error can't find the container with id f73e959eb789e07be29377c25fdaadbe86b0ae35e89ebbfd667a0a555fce492a
Jul 02 13:20:14.489672 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:14.489638    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6bbd58757b-88495"]
Jul 02 13:20:14.491739 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:20:14.491715    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38758751_9e67_4e38_847f_e0c26baa24dd.slice/crio-f3fe0546b85fa8d1c7ddb2a4a52886907c42646a564dda817d59c73eeeccda6d WatchSource:0}: Error finding container f3fe0546b85fa8d1c7ddb2a4a52886907c42646a564dda817d59c73eeeccda6d: Status 404 returned error can't find the container with id f3fe0546b85fa8d1c7ddb2a4a52886907c42646a564dda817d59c73eeeccda6d
Jul 02 13:20:15.283018 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:15.282930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj" event={"ID":"8e30eb35-606a-45ba-912d-7929ad10b3f3","Type":"ContainerStarted","Data":"f73e959eb789e07be29377c25fdaadbe86b0ae35e89ebbfd667a0a555fce492a"}
Jul 02 13:20:15.284833 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:15.284799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495" event={"ID":"38758751-9e67-4e38-847f-e0c26baa24dd","Type":"ContainerStarted","Data":"f3fe0546b85fa8d1c7ddb2a4a52886907c42646a564dda817d59c73eeeccda6d"}
Jul 02 13:20:19.303121 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:19.303080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495" event={"ID":"38758751-9e67-4e38-847f-e0c26baa24dd","Type":"ContainerStarted","Data":"e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df"}
Jul 02 13:20:19.303610 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:19.303193    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:19.304599 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:19.304577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj" event={"ID":"8e30eb35-606a-45ba-912d-7929ad10b3f3","Type":"ContainerStarted","Data":"df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d"}
Jul 02 13:20:19.304720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:19.304709    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:19.337688 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:19.337636    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495" podStartSLOduration=1.251732778 podStartE2EDuration="5.337620644s" totalImagesPullingTime="4.085887866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:20:19.336740775 +0000 UTC m=+394.067778900" watchObservedRunningTime="2026-07-02 13:20:19.337620644 +0000 UTC m=+394.068658763"
Jul 02 13:20:19.362480 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:19.362425    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj" podStartSLOduration=2.257919668 podStartE2EDuration="6.362408646s" totalImagesPullingTime="4.104488978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:20:19.361169386 +0000 UTC m=+394.092207537" watchObservedRunningTime="2026-07-02 13:20:19.362408646 +0000 UTC m=+394.093447055"
Jul 02 13:20:50.310440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:50.310408    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 13:20:50.313637 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:50.313615    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:20:51.698345 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.698311    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"]
Jul 02 13:20:51.761630 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.761592    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.764943 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.764913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:20:51.765100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.764989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lktlz\""
Jul 02 13:20:51.766445 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.766423    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:20:51.767911 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.767891    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"]
Jul 02 13:20:51.847830 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.847799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.847972 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.847837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.847972 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.847862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24896\" (UniqueName: \"kubernetes.io/projected/9aa95c3d-60c5-4500-a2a1-25440645530f-kube-api-access-24896\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.948853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.948774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.948853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.948832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24896\" (UniqueName: \"kubernetes.io/projected/9aa95c3d-60c5-4500-a2a1-25440645530f-kube-api-access-24896\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.949046 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.948896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.949165 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.949144    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.949229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.949170    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:51.958976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:51.958956    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24896\" (UniqueName: \"kubernetes.io/projected/9aa95c3d-60c5-4500-a2a1-25440645530f-kube-api-access-24896\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:52.071811 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:52.071778    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:52.229402 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:52.229376    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"]
Jul 02 13:20:52.231256 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:20:52.231226    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa95c3d_60c5_4500_a2a1_25440645530f.slice/crio-8af65f320fce12cdb72b2dc592196a64805bf47c23d799013df5f229f74d2f3e WatchSource:0}: Error finding container 8af65f320fce12cdb72b2dc592196a64805bf47c23d799013df5f229f74d2f3e: Status 404 returned error can't find the container with id 8af65f320fce12cdb72b2dc592196a64805bf47c23d799013df5f229f74d2f3e
Jul 02 13:20:52.441293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:52.441256    2564 generic.go:356] "Generic (PLEG): container finished" podID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerID="6b314a8b6cc0ce363de2fc0963e87aa82e72c8194aa533947bd4c5292d40f176" exitCode=0
Jul 02 13:20:52.441448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:52.441341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb" event={"ID":"9aa95c3d-60c5-4500-a2a1-25440645530f","Type":"ContainerDied","Data":"6b314a8b6cc0ce363de2fc0963e87aa82e72c8194aa533947bd4c5292d40f176"}
Jul 02 13:20:52.441448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:52.441393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb" event={"ID":"9aa95c3d-60c5-4500-a2a1-25440645530f","Type":"ContainerStarted","Data":"8af65f320fce12cdb72b2dc592196a64805bf47c23d799013df5f229f74d2f3e"}
Jul 02 13:20:55.459323 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:55.459044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb" event={"ID":"9aa95c3d-60c5-4500-a2a1-25440645530f","Type":"ContainerStarted","Data":"aa2d6e24656077405b8aca8aed96b13d94772a82522fc5674c57cd4f8f9cd8b6"}
Jul 02 13:20:56.465361 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:56.465324    2564 generic.go:356] "Generic (PLEG): container finished" podID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerID="aa2d6e24656077405b8aca8aed96b13d94772a82522fc5674c57cd4f8f9cd8b6" exitCode=0
Jul 02 13:20:56.465781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:56.465369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb" event={"ID":"9aa95c3d-60c5-4500-a2a1-25440645530f","Type":"ContainerDied","Data":"aa2d6e24656077405b8aca8aed96b13d94772a82522fc5674c57cd4f8f9cd8b6"}
Jul 02 13:20:57.471374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:57.471336    2564 generic.go:356] "Generic (PLEG): container finished" podID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerID="91a6212166cc9b25c1d0457ae6fbf12104cf5a626b31e260e722c5fb29875588" exitCode=0
Jul 02 13:20:57.471778 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:57.471425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb" event={"ID":"9aa95c3d-60c5-4500-a2a1-25440645530f","Type":"ContainerDied","Data":"91a6212166cc9b25c1d0457ae6fbf12104cf5a626b31e260e722c5fb29875588"}
Jul 02 13:20:58.609541 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.609498    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:58.704770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.704736    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-bundle\") pod \"9aa95c3d-60c5-4500-a2a1-25440645530f\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") "
Jul 02 13:20:58.704957 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.704794    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-util\") pod \"9aa95c3d-60c5-4500-a2a1-25440645530f\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") "
Jul 02 13:20:58.704957 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.704814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24896\" (UniqueName: \"kubernetes.io/projected/9aa95c3d-60c5-4500-a2a1-25440645530f-kube-api-access-24896\") pod \"9aa95c3d-60c5-4500-a2a1-25440645530f\" (UID: \"9aa95c3d-60c5-4500-a2a1-25440645530f\") "
Jul 02 13:20:58.705546 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.705502    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-bundle" (OuterVolumeSpecName: "bundle") pod "9aa95c3d-60c5-4500-a2a1-25440645530f" (UID: "9aa95c3d-60c5-4500-a2a1-25440645530f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:20:58.706957 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.706935    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa95c3d-60c5-4500-a2a1-25440645530f-kube-api-access-24896" (OuterVolumeSpecName: "kube-api-access-24896") pod "9aa95c3d-60c5-4500-a2a1-25440645530f" (UID: "9aa95c3d-60c5-4500-a2a1-25440645530f"). InnerVolumeSpecName "kube-api-access-24896". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:20:58.709602 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.709573    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-util" (OuterVolumeSpecName: "util") pod "9aa95c3d-60c5-4500-a2a1-25440645530f" (UID: "9aa95c3d-60c5-4500-a2a1-25440645530f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:20:58.806316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.806228    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:20:58.806316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.806264    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24896\" (UniqueName: \"kubernetes.io/projected/9aa95c3d-60c5-4500-a2a1-25440645530f-kube-api-access-24896\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:20:58.806316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:58.806275    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9aa95c3d-60c5-4500-a2a1-25440645530f-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:20:59.481822 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:59.481793    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb"
Jul 02 13:20:59.482003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:59.481787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dg5qhb" event={"ID":"9aa95c3d-60c5-4500-a2a1-25440645530f","Type":"ContainerDied","Data":"8af65f320fce12cdb72b2dc592196a64805bf47c23d799013df5f229f74d2f3e"}
Jul 02 13:20:59.482003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:20:59.481908    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af65f320fce12cdb72b2dc592196a64805bf47c23d799013df5f229f74d2f3e"
Jul 02 13:21:23.929947 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.929867    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"]
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930195    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="extract"
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930207    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="extract"
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930220    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="util"
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930225    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="util"
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930233    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="pull"
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930238    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="pull"
Jul 02 13:21:23.930432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.930292    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa95c3d-60c5-4500-a2a1-25440645530f" containerName="extract"
Jul 02 13:21:23.933468 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.933451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:23.936647 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.936623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:21:23.936775 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.936688    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:21:23.937996 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.937971    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lktlz\""
Jul 02 13:21:23.944399 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.944378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"]
Jul 02 13:21:23.979898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.979870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vww\" (UniqueName: \"kubernetes.io/projected/c51eda7e-356f-42eb-9095-92f04d964877-kube-api-access-76vww\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:23.979898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.979906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:23.980097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:23.979946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.081213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.081172    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.081395 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.081253    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vww\" (UniqueName: \"kubernetes.io/projected/c51eda7e-356f-42eb-9095-92f04d964877-kube-api-access-76vww\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.081395 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.081278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.081707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.081685    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.081769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.081688    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.090346 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.090324    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vww\" (UniqueName: \"kubernetes.io/projected/c51eda7e-356f-42eb-9095-92f04d964877-kube-api-access-76vww\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.242735 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.242650    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:24.377216 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.377187    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"]
Jul 02 13:21:24.379658 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:21:24.379629    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51eda7e_356f_42eb_9095_92f04d964877.slice/crio-78f0ecf609db140639626cee492052793de9f5c6d86dedeb563f96a192914fbd WatchSource:0}: Error finding container 78f0ecf609db140639626cee492052793de9f5c6d86dedeb563f96a192914fbd: Status 404 returned error can't find the container with id 78f0ecf609db140639626cee492052793de9f5c6d86dedeb563f96a192914fbd
Jul 02 13:21:24.588998 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.588916    2564 generic.go:356] "Generic (PLEG): container finished" podID="c51eda7e-356f-42eb-9095-92f04d964877" containerID="195ddae290130de70caf867f0e26b45f2a4c8ebd65eaf71af28b60e332da9ba9" exitCode=0
Jul 02 13:21:24.589142 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.588999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s" event={"ID":"c51eda7e-356f-42eb-9095-92f04d964877","Type":"ContainerDied","Data":"195ddae290130de70caf867f0e26b45f2a4c8ebd65eaf71af28b60e332da9ba9"}
Jul 02 13:21:24.589142 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:24.589051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s" event={"ID":"c51eda7e-356f-42eb-9095-92f04d964877","Type":"ContainerStarted","Data":"78f0ecf609db140639626cee492052793de9f5c6d86dedeb563f96a192914fbd"}
Jul 02 13:21:26.598225 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.598189    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86459b6747-h2gl2"]
Jul 02 13:21:26.607297 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.607270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.624133 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.624103    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86459b6747-h2gl2"]
Jul 02 13:21:26.703670 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-service-ca\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.703854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703680    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkxr\" (UniqueName: \"kubernetes.io/projected/46c94572-6b06-42b4-b411-71759517e639-kube-api-access-7tkxr\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.703854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46c94572-6b06-42b4-b411-71759517e639-console-oauth-config\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.703854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-trusted-ca-bundle\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.703854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-console-config\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.704072 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46c94572-6b06-42b4-b411-71759517e639-console-serving-cert\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.704072 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.703992    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-oauth-serving-cert\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805087 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-service-ca\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkxr\" (UniqueName: \"kubernetes.io/projected/46c94572-6b06-42b4-b411-71759517e639-kube-api-access-7tkxr\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46c94572-6b06-42b4-b411-71759517e639-console-oauth-config\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-trusted-ca-bundle\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-console-config\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46c94572-6b06-42b4-b411-71759517e639-console-serving-cert\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805499 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-oauth-serving-cert\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.805924 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.805860    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-service-ca\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.806069 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.806008    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-oauth-serving-cert\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.806134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.806111    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-trusted-ca-bundle\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.806197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.806111    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46c94572-6b06-42b4-b411-71759517e639-console-config\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.808030 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.808004    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46c94572-6b06-42b4-b411-71759517e639-console-oauth-config\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.808217 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.808193    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46c94572-6b06-42b4-b411-71759517e639-console-serving-cert\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.815418 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.815390    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkxr\" (UniqueName: \"kubernetes.io/projected/46c94572-6b06-42b4-b411-71759517e639-kube-api-access-7tkxr\") pod \"console-86459b6747-h2gl2\" (UID: \"46c94572-6b06-42b4-b411-71759517e639\") " pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:26.917935 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:26.917886    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:27.135247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:27.135220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86459b6747-h2gl2"]
Jul 02 13:21:27.138166 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:21:27.138140    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c94572_6b06_42b4_b411_71759517e639.slice/crio-1b76a885c9e47eb6fab18835494caf4377a71520064980b39fc53e37a82abeaa WatchSource:0}: Error finding container 1b76a885c9e47eb6fab18835494caf4377a71520064980b39fc53e37a82abeaa: Status 404 returned error can't find the container with id 1b76a885c9e47eb6fab18835494caf4377a71520064980b39fc53e37a82abeaa
Jul 02 13:21:27.608293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:27.608200    2564 generic.go:356] "Generic (PLEG): container finished" podID="c51eda7e-356f-42eb-9095-92f04d964877" containerID="a62ad8ea6a9949adf3f7df5a6123af68513443657c90d4891fae845e54f8a11d" exitCode=0
Jul 02 13:21:27.608293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:27.608280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s" event={"ID":"c51eda7e-356f-42eb-9095-92f04d964877","Type":"ContainerDied","Data":"a62ad8ea6a9949adf3f7df5a6123af68513443657c90d4891fae845e54f8a11d"}
Jul 02 13:21:27.609980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:27.609960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86459b6747-h2gl2" event={"ID":"46c94572-6b06-42b4-b411-71759517e639","Type":"ContainerStarted","Data":"7363df03762c20c28a3f6090f87d9cb2f4c07db7e490de129294800216ece324"}
Jul 02 13:21:27.610054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:27.610003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86459b6747-h2gl2" event={"ID":"46c94572-6b06-42b4-b411-71759517e639","Type":"ContainerStarted","Data":"1b76a885c9e47eb6fab18835494caf4377a71520064980b39fc53e37a82abeaa"}
Jul 02 13:21:27.647972 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:27.647930    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86459b6747-h2gl2" podStartSLOduration=1.647914995 podStartE2EDuration="1.647914995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:27.646950915 +0000 UTC m=+462.377989067" watchObservedRunningTime="2026-07-02 13:21:27.647914995 +0000 UTC m=+462.378953202"
Jul 02 13:21:28.616090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:28.616050    2564 generic.go:356] "Generic (PLEG): container finished" podID="c51eda7e-356f-42eb-9095-92f04d964877" containerID="47c186ff32029524a241d3108f80bf346e4e432e9dba3395c8214dca0e027550" exitCode=0
Jul 02 13:21:28.616582 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:28.616135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s" event={"ID":"c51eda7e-356f-42eb-9095-92f04d964877","Type":"ContainerDied","Data":"47c186ff32029524a241d3108f80bf346e4e432e9dba3395c8214dca0e027550"}
Jul 02 13:21:29.755083 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.755057    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:29.830338 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.830302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-util\") pod \"c51eda7e-356f-42eb-9095-92f04d964877\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") "
Jul 02 13:21:29.830338 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.830349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-bundle\") pod \"c51eda7e-356f-42eb-9095-92f04d964877\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") "
Jul 02 13:21:29.830598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.830380    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76vww\" (UniqueName: \"kubernetes.io/projected/c51eda7e-356f-42eb-9095-92f04d964877-kube-api-access-76vww\") pod \"c51eda7e-356f-42eb-9095-92f04d964877\" (UID: \"c51eda7e-356f-42eb-9095-92f04d964877\") "
Jul 02 13:21:29.830774 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.830752    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-bundle" (OuterVolumeSpecName: "bundle") pod "c51eda7e-356f-42eb-9095-92f04d964877" (UID: "c51eda7e-356f-42eb-9095-92f04d964877"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:21:29.832539 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.832498    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51eda7e-356f-42eb-9095-92f04d964877-kube-api-access-76vww" (OuterVolumeSpecName: "kube-api-access-76vww") pod "c51eda7e-356f-42eb-9095-92f04d964877" (UID: "c51eda7e-356f-42eb-9095-92f04d964877"). InnerVolumeSpecName "kube-api-access-76vww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:21:29.834730 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.834699    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-util" (OuterVolumeSpecName: "util") pod "c51eda7e-356f-42eb-9095-92f04d964877" (UID: "c51eda7e-356f-42eb-9095-92f04d964877"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:21:29.931605 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.931574    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:21:29.931605 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.931602    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c51eda7e-356f-42eb-9095-92f04d964877-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:21:29.931605 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:29.931613    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vww\" (UniqueName: \"kubernetes.io/projected/c51eda7e-356f-42eb-9095-92f04d964877-kube-api-access-76vww\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:21:30.626930 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:30.626894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s" event={"ID":"c51eda7e-356f-42eb-9095-92f04d964877","Type":"ContainerDied","Data":"78f0ecf609db140639626cee492052793de9f5c6d86dedeb563f96a192914fbd"}
Jul 02 13:21:30.627102 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:30.626948    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9bp8s"
Jul 02 13:21:30.627102 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:30.626953    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f0ecf609db140639626cee492052793de9f5c6d86dedeb563f96a192914fbd"
Jul 02 13:21:36.918746 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:36.918707    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:36.919121 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:36.918775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:36.923705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:36.923683    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:37.661298 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:37.661271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86459b6747-h2gl2"
Jul 02 13:21:37.728411 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:37.728379    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9bbc5b66-xqbfn"]
Jul 02 13:21:47.370698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.370662    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"]
Jul 02 13:21:47.371056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.370993    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="pull"
Jul 02 13:21:47.371056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.371006    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="pull"
Jul 02 13:21:47.371056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.371021    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="util"
Jul 02 13:21:47.371056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.371026    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="util"
Jul 02 13:21:47.371056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.371038    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="extract"
Jul 02 13:21:47.371056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.371043    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="extract"
Jul 02 13:21:47.371292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.371107    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51eda7e-356f-42eb-9095-92f04d964877" containerName="extract"
Jul 02 13:21:47.374218 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.374201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.379060 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.379038    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 02 13:21:47.379182 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.379082    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 02 13:21:47.379182 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.379041    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 02 13:21:47.379435 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.379416    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 02 13:21:47.379510 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.379441    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mwnhw\""
Jul 02 13:21:47.388793 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.388769    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"]
Jul 02 13:21:47.478237 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.478385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.478385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.478385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzj9\" (UniqueName: \"kubernetes.io/projected/23b14cdb-03b2-444e-9412-2db9b25c1614-kube-api-access-jhzj9\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.478385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.478385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b14cdb-03b2-444e-9412-2db9b25c1614-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.478577 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.478399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.579712 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.579871 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.579871 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.579953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.579953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzj9\" (UniqueName: \"kubernetes.io/projected/23b14cdb-03b2-444e-9412-2db9b25c1614-kube-api-access-jhzj9\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.579953 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.580100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.579971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b14cdb-03b2-444e-9412-2db9b25c1614-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.580727 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.580696    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.582241 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.582216    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.582666 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.582637    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b14cdb-03b2-444e-9412-2db9b25c1614-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.582766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.582641    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.582766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.582682    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23b14cdb-03b2-444e-9412-2db9b25c1614-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.589035 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.589011    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b14cdb-03b2-444e-9412-2db9b25c1614-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.589234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.589218    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzj9\" (UniqueName: \"kubernetes.io/projected/23b14cdb-03b2-444e-9412-2db9b25c1614-kube-api-access-jhzj9\") pod \"istiod-openshift-gateway-94bb8fbfd-68rqc\" (UID: \"23b14cdb-03b2-444e-9412-2db9b25c1614\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.683319 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.683287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:47.826960 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:47.826931    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"]
Jul 02 13:21:47.828437 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:21:47.828405    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b14cdb_03b2_444e_9412_2db9b25c1614.slice/crio-a9dd605b52d65073827fba6b4e37a703526728336a42187d9a7f82332fb82138 WatchSource:0}: Error finding container a9dd605b52d65073827fba6b4e37a703526728336a42187d9a7f82332fb82138: Status 404 returned error can't find the container with id a9dd605b52d65073827fba6b4e37a703526728336a42187d9a7f82332fb82138
Jul 02 13:21:48.705828 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:48.705788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc" event={"ID":"23b14cdb-03b2-444e-9412-2db9b25c1614","Type":"ContainerStarted","Data":"a9dd605b52d65073827fba6b4e37a703526728336a42187d9a7f82332fb82138"}
Jul 02 13:21:51.695180 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:51.695122    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 02 13:21:51.695484 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:51.695193    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 02 13:21:52.734179 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:52.734130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc" event={"ID":"23b14cdb-03b2-444e-9412-2db9b25c1614","Type":"ContainerStarted","Data":"409d3edff275c0b5c73b1d20470de6e7231cfdddd4425f703cf8c64c90f64963"}
Jul 02 13:21:52.734608 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:52.734347    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:21:52.735976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:52.735946    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-68rqc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 02 13:21:52.736128 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:52.736019    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc" podUID="23b14cdb-03b2-444e-9412-2db9b25c1614" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:21:52.760497 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:52.760452    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc" podStartSLOduration=1.89606425 podStartE2EDuration="5.760436706s" totalImagesPullingTime="3.864372456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:52.758080957 +0000 UTC m=+487.489119107" watchObservedRunningTime="2026-07-02 13:21:52.760436706 +0000 UTC m=+487.491474847"
Jul 02 13:21:53.739386 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:21:53.739356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc"
Jul 02 13:22:01.786018 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.785986    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"]
Jul 02 13:22:01.793828 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.793794    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:01.796705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.796674    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:22:01.796881 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.796718    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:22:01.798825 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.798799    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lktlz\""
Jul 02 13:22:01.800711 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.800691    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"]
Jul 02 13:22:01.887090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.887059    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"]
Jul 02 13:22:01.890843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.890826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:01.899848 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.899818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"]
Jul 02 13:22:01.909230 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.909198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:01.909472 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.909248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drhl\" (UniqueName: \"kubernetes.io/projected/d80a5891-8573-45e1-a741-896aa134f92c-kube-api-access-4drhl\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:01.909472 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.909333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:01.989002 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.988963    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"]
Jul 02 13:22:01.994201 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.994175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:01.998543 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:01.998500    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"]
Jul 02 13:22:02.010431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.010690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.010690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.010690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4sdd\" (UniqueName: \"kubernetes.io/projected/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-kube-api-access-h4sdd\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.010690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.010690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drhl\" (UniqueName: \"kubernetes.io/projected/d80a5891-8573-45e1-a741-896aa134f92c-kube-api-access-4drhl\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.011052 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.010919    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.011340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.011303    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.019054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.019034    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drhl\" (UniqueName: \"kubernetes.io/projected/d80a5891-8573-45e1-a741-896aa134f92c-kube-api-access-4drhl\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.090707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.090627    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"]
Jul 02 13:22:02.094498 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.094474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.103901 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.103879    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"]
Jul 02 13:22:02.104980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.104957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:02.111146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.111262 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4sdd\" (UniqueName: \"kubernetes.io/projected/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-kube-api-access-h4sdd\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.111262 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.111353 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.111353 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbd5\" (UniqueName: \"kubernetes.io/projected/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-kube-api-access-txbd5\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.111422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.111539 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111502    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.111737 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.111706    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.121248 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.121221    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4sdd\" (UniqueName: \"kubernetes.io/projected/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-kube-api-access-h4sdd\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.200894 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.200845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:02.212900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.212863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.213065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w579v\" (UniqueName: \"kubernetes.io/projected/726977eb-d563-4182-b3d7-bcaea887084a-kube-api-access-w579v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.213132 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.213190 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213163    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.213249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.213302 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbd5\" (UniqueName: \"kubernetes.io/projected/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-kube-api-access-txbd5\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.213469 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213447    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.213809 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.213785    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.223138 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.223105    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbd5\" (UniqueName: \"kubernetes.io/projected/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-kube-api-access-txbd5\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.245198 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.245160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"]
Jul 02 13:22:02.248498 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:02.248120    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80a5891_8573_45e1_a741_896aa134f92c.slice/crio-8c0771d3ba1542f2691702f4b4f69ec41ed4ac8a42ed91e4e8615a3b8470b481 WatchSource:0}: Error finding container 8c0771d3ba1542f2691702f4b4f69ec41ed4ac8a42ed91e4e8615a3b8470b481: Status 404 returned error can't find the container with id 8c0771d3ba1542f2691702f4b4f69ec41ed4ac8a42ed91e4e8615a3b8470b481
Jul 02 13:22:02.304739 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.304711    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:02.314834 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.314589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.314834 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.314680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.315000 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.314860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w579v\" (UniqueName: \"kubernetes.io/projected/726977eb-d563-4182-b3d7-bcaea887084a-kube-api-access-w579v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.315589 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.315562    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.315917 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.315871    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.324223 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.324200    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w579v\" (UniqueName: \"kubernetes.io/projected/726977eb-d563-4182-b3d7-bcaea887084a-kube-api-access-w579v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.347706 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.347678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"]
Jul 02 13:22:02.349739 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:02.349705    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79ecd78_9893_43a0_81fe_ca9f6cd3149d.slice/crio-589917868f0f9354c2b84f73d9d1f431c1c69374c26f77a078aa96f64b0f2e89 WatchSource:0}: Error finding container 589917868f0f9354c2b84f73d9d1f431c1c69374c26f77a078aa96f64b0f2e89: Status 404 returned error can't find the container with id 589917868f0f9354c2b84f73d9d1f431c1c69374c26f77a078aa96f64b0f2e89
Jul 02 13:22:02.404860 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.404834    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:02.451203 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.450858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"]
Jul 02 13:22:02.453053 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:02.453023    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b5d3d3_4326_4aa4_9bca_2df6b2d20dc3.slice/crio-02835cd8661d3fc8442dbb53f854e15ea61b9210ae39e152b15070fd1c394b3d WatchSource:0}: Error finding container 02835cd8661d3fc8442dbb53f854e15ea61b9210ae39e152b15070fd1c394b3d: Status 404 returned error can't find the container with id 02835cd8661d3fc8442dbb53f854e15ea61b9210ae39e152b15070fd1c394b3d
Jul 02 13:22:02.557676 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.557651    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"]
Jul 02 13:22:02.617335 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:02.617273    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726977eb_d563_4182_b3d7_bcaea887084a.slice/crio-f6b80726771d10de8e93ba72d7ec17ce7c6cf3c776354c489c74fd766188807a WatchSource:0}: Error finding container f6b80726771d10de8e93ba72d7ec17ce7c6cf3c776354c489c74fd766188807a: Status 404 returned error can't find the container with id f6b80726771d10de8e93ba72d7ec17ce7c6cf3c776354c489c74fd766188807a
Jul 02 13:22:02.751252 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.751206    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-d9bbc5b66-xqbfn" podUID="cb5fce27-47f9-4546-ab87-2f03ac191068" containerName="console" containerID="cri-o://37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2" gracePeriod=15
Jul 02 13:22:02.778564 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.778510    2564 generic.go:356] "Generic (PLEG): container finished" podID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerID="0013c9a16c1a9c829d307a0f94f925e417eb90fa7bfa5ac5efd82918418ce3be" exitCode=0
Jul 02 13:22:02.778676 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.778565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj" event={"ID":"c79ecd78-9893-43a0-81fe-ca9f6cd3149d","Type":"ContainerDied","Data":"0013c9a16c1a9c829d307a0f94f925e417eb90fa7bfa5ac5efd82918418ce3be"}
Jul 02 13:22:02.778676 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.778620    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj" event={"ID":"c79ecd78-9893-43a0-81fe-ca9f6cd3149d","Type":"ContainerStarted","Data":"589917868f0f9354c2b84f73d9d1f431c1c69374c26f77a078aa96f64b0f2e89"}
Jul 02 13:22:02.780231 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.780208    2564 generic.go:356] "Generic (PLEG): container finished" podID="d80a5891-8573-45e1-a741-896aa134f92c" containerID="1fb1ea13431fdbecf5cc6db233ae2f82f06809f154524c02c5b8eceaa523cb3f" exitCode=0
Jul 02 13:22:02.780327 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.780260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr" event={"ID":"d80a5891-8573-45e1-a741-896aa134f92c","Type":"ContainerDied","Data":"1fb1ea13431fdbecf5cc6db233ae2f82f06809f154524c02c5b8eceaa523cb3f"}
Jul 02 13:22:02.780327 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.780306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr" event={"ID":"d80a5891-8573-45e1-a741-896aa134f92c","Type":"ContainerStarted","Data":"8c0771d3ba1542f2691702f4b4f69ec41ed4ac8a42ed91e4e8615a3b8470b481"}
Jul 02 13:22:02.781901 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.781879    2564 generic.go:356] "Generic (PLEG): container finished" podID="726977eb-d563-4182-b3d7-bcaea887084a" containerID="394af2ec7ea476174f5f8609543cdc0514b607ff5e7fc3e67ab239862b7a497c" exitCode=0
Jul 02 13:22:02.781964 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.781906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6" event={"ID":"726977eb-d563-4182-b3d7-bcaea887084a","Type":"ContainerDied","Data":"394af2ec7ea476174f5f8609543cdc0514b607ff5e7fc3e67ab239862b7a497c"}
Jul 02 13:22:02.781964 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.781946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6" event={"ID":"726977eb-d563-4182-b3d7-bcaea887084a","Type":"ContainerStarted","Data":"f6b80726771d10de8e93ba72d7ec17ce7c6cf3c776354c489c74fd766188807a"}
Jul 02 13:22:02.783704 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.783585    2564 generic.go:356] "Generic (PLEG): container finished" podID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerID="324174a55bdd9db5855a5ef14b28b6410267d99d8127cd970eb00b81a048e0a0" exitCode=0
Jul 02 13:22:02.783704 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.783637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6" event={"ID":"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3","Type":"ContainerDied","Data":"324174a55bdd9db5855a5ef14b28b6410267d99d8127cd970eb00b81a048e0a0"}
Jul 02 13:22:02.783704 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.783694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6" event={"ID":"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3","Type":"ContainerStarted","Data":"02835cd8661d3fc8442dbb53f854e15ea61b9210ae39e152b15070fd1c394b3d"}
Jul 02 13:22:02.996233 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:02.996208    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:22:03.122794 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-oauth-config\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.122794 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122760    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-trusted-ca-bundle\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.122794 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122778    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-console-config\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.123049 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-oauth-serving-cert\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.123049 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-service-ca\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.123049 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbc45\" (UniqueName: \"kubernetes.io/projected/cb5fce27-47f9-4546-ab87-2f03ac191068-kube-api-access-hbc45\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.123049 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.122941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-serving-cert\") pod \"cb5fce27-47f9-4546-ab87-2f03ac191068\" (UID: \"cb5fce27-47f9-4546-ab87-2f03ac191068\") "
Jul 02 13:22:03.123280 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.123252    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-service-ca" (OuterVolumeSpecName: "service-ca") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:22:03.123280 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.123264    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:22:03.123397 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.123280    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-console-config" (OuterVolumeSpecName: "console-config") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:22:03.123397 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.123323    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:22:03.125021 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.124986    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:22:03.125129 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.125064    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:22:03.125129 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.125114    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5fce27-47f9-4546-ab87-2f03ac191068-kube-api-access-hbc45" (OuterVolumeSpecName: "kube-api-access-hbc45") pod "cb5fce27-47f9-4546-ab87-2f03ac191068" (UID: "cb5fce27-47f9-4546-ab87-2f03ac191068"). InnerVolumeSpecName "kube-api-access-hbc45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:03.224573 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224521    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-trusted-ca-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.224573 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224567    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-console-config\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.224573 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224576    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-oauth-serving-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.224782 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224584    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5fce27-47f9-4546-ab87-2f03ac191068-service-ca\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.224782 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224594    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbc45\" (UniqueName: \"kubernetes.io/projected/cb5fce27-47f9-4546-ab87-2f03ac191068-kube-api-access-hbc45\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.224782 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224602    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-serving-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.224782 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.224610    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb5fce27-47f9-4546-ab87-2f03ac191068-console-oauth-config\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:03.791964 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.791935    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb5fce27-47f9-4546-ab87-2f03ac191068" containerID="37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2" exitCode=0
Jul 02 13:22:03.792088 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.791992    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bbc5b66-xqbfn"
Jul 02 13:22:03.792088 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.792019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bbc5b66-xqbfn" event={"ID":"cb5fce27-47f9-4546-ab87-2f03ac191068","Type":"ContainerDied","Data":"37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2"}
Jul 02 13:22:03.792088 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.792070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bbc5b66-xqbfn" event={"ID":"cb5fce27-47f9-4546-ab87-2f03ac191068","Type":"ContainerDied","Data":"845436cb837782e564a98fbb5df29c1e0015c55db060b444c7b10e5e5b0ab2f2"}
Jul 02 13:22:03.792250 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.792098    2564 scope.go:117] "RemoveContainer" containerID="37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2"
Jul 02 13:22:03.800923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.800903    2564 scope.go:117] "RemoveContainer" containerID="37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2"
Jul 02 13:22:03.801186 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:22:03.801167    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2\": container with ID starting with 37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2 not found: ID does not exist" containerID="37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2"
Jul 02 13:22:03.801267 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.801196    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2"} err="failed to get container status \"37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2\": rpc error: code = NotFound desc = could not find container \"37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2\": container with ID starting with 37548dd5ad5ec7e8ce21b6c8c71d673f98fb342ac557bc9a079a03733eb07ca2 not found: ID does not exist"
Jul 02 13:22:03.813711 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.813684    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9bbc5b66-xqbfn"]
Jul 02 13:22:03.823807 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:03.823785    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d9bbc5b66-xqbfn"]
Jul 02 13:22:04.801045 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.801010    2564 generic.go:356] "Generic (PLEG): container finished" podID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerID="224f833480fab96256a36c7b0bb395725b5e6084c882b72bc36fc03844bc9ef2" exitCode=0
Jul 02 13:22:04.801477 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.801096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj" event={"ID":"c79ecd78-9893-43a0-81fe-ca9f6cd3149d","Type":"ContainerDied","Data":"224f833480fab96256a36c7b0bb395725b5e6084c882b72bc36fc03844bc9ef2"}
Jul 02 13:22:04.803183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.803162    2564 generic.go:356] "Generic (PLEG): container finished" podID="d80a5891-8573-45e1-a741-896aa134f92c" containerID="100117cf905dbc211cc6af336a05b625109ace2d88ee5843c9967b5f69ab981f" exitCode=0
Jul 02 13:22:04.803249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.803225    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr" event={"ID":"d80a5891-8573-45e1-a741-896aa134f92c","Type":"ContainerDied","Data":"100117cf905dbc211cc6af336a05b625109ace2d88ee5843c9967b5f69ab981f"}
Jul 02 13:22:04.805115 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.805095    2564 generic.go:356] "Generic (PLEG): container finished" podID="726977eb-d563-4182-b3d7-bcaea887084a" containerID="533e1e74028e17f00b9a8d65d7d072cf98e33115bd5605dafd3aa1f405e480df" exitCode=0
Jul 02 13:22:04.805195 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.805162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6" event={"ID":"726977eb-d563-4182-b3d7-bcaea887084a","Type":"ContainerDied","Data":"533e1e74028e17f00b9a8d65d7d072cf98e33115bd5605dafd3aa1f405e480df"}
Jul 02 13:22:04.810647 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.810622    2564 generic.go:356] "Generic (PLEG): container finished" podID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerID="a1bf29849eb7f8bded1032711b55d3d2d74f77408a96372e5c7535f3baffd921" exitCode=0
Jul 02 13:22:04.810768 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:04.810744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6" event={"ID":"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3","Type":"ContainerDied","Data":"a1bf29849eb7f8bded1032711b55d3d2d74f77408a96372e5c7535f3baffd921"}
Jul 02 13:22:05.760418 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.760382    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5fce27-47f9-4546-ab87-2f03ac191068" path="/var/lib/kubelet/pods/cb5fce27-47f9-4546-ab87-2f03ac191068/volumes"
Jul 02 13:22:05.818946 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.818912    2564 generic.go:356] "Generic (PLEG): container finished" podID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerID="f68608da51158f56037f34dc08aa7724e63c33dc4e5f6db4bf18e445b80ada45" exitCode=0
Jul 02 13:22:05.819350 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.819004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj" event={"ID":"c79ecd78-9893-43a0-81fe-ca9f6cd3149d","Type":"ContainerDied","Data":"f68608da51158f56037f34dc08aa7724e63c33dc4e5f6db4bf18e445b80ada45"}
Jul 02 13:22:05.820881 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.820860    2564 generic.go:356] "Generic (PLEG): container finished" podID="d80a5891-8573-45e1-a741-896aa134f92c" containerID="a5310f48c923486242254d36f79ce70b682d8a302f89d3f09408fafeea122bf9" exitCode=0
Jul 02 13:22:05.821000 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.820938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr" event={"ID":"d80a5891-8573-45e1-a741-896aa134f92c","Type":"ContainerDied","Data":"a5310f48c923486242254d36f79ce70b682d8a302f89d3f09408fafeea122bf9"}
Jul 02 13:22:05.822777 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.822757    2564 generic.go:356] "Generic (PLEG): container finished" podID="726977eb-d563-4182-b3d7-bcaea887084a" containerID="ddc9e742e6a8eb415a46e8fe86a4b11e15472b2db83366f4358f5df1b361724c" exitCode=0
Jul 02 13:22:05.822866 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.822819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6" event={"ID":"726977eb-d563-4182-b3d7-bcaea887084a","Type":"ContainerDied","Data":"ddc9e742e6a8eb415a46e8fe86a4b11e15472b2db83366f4358f5df1b361724c"}
Jul 02 13:22:05.824740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.824722    2564 generic.go:356] "Generic (PLEG): container finished" podID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerID="c2b27afffe361768cae8de58bfc1b4eb965d6cb4a6d9ee5cb210217e71d96a12" exitCode=0
Jul 02 13:22:05.824824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:05.824752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6" event={"ID":"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3","Type":"ContainerDied","Data":"c2b27afffe361768cae8de58bfc1b4eb965d6cb4a6d9ee5cb210217e71d96a12"}
Jul 02 13:22:06.989993 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:06.989960    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:06.992113 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:06.992088    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:07.074132 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.074106    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:07.076848 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.076825    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:07.158646 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.158614    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-util\") pod \"d80a5891-8573-45e1-a741-896aa134f92c\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") "
Jul 02 13:22:07.158841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.158660    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4sdd\" (UniqueName: \"kubernetes.io/projected/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-kube-api-access-h4sdd\") pod \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") "
Jul 02 13:22:07.158841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.158698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-bundle\") pod \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") "
Jul 02 13:22:07.158841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.158715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-util\") pod \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\" (UID: \"c79ecd78-9893-43a0-81fe-ca9f6cd3149d\") "
Jul 02 13:22:07.158841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.158734    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-bundle\") pod \"d80a5891-8573-45e1-a741-896aa134f92c\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") "
Jul 02 13:22:07.158841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.158774    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4drhl\" (UniqueName: \"kubernetes.io/projected/d80a5891-8573-45e1-a741-896aa134f92c-kube-api-access-4drhl\") pod \"d80a5891-8573-45e1-a741-896aa134f92c\" (UID: \"d80a5891-8573-45e1-a741-896aa134f92c\") "
Jul 02 13:22:07.159331 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.159281    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-bundle" (OuterVolumeSpecName: "bundle") pod "d80a5891-8573-45e1-a741-896aa134f92c" (UID: "d80a5891-8573-45e1-a741-896aa134f92c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.159445 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.159331    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-bundle" (OuterVolumeSpecName: "bundle") pod "c79ecd78-9893-43a0-81fe-ca9f6cd3149d" (UID: "c79ecd78-9893-43a0-81fe-ca9f6cd3149d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.160952 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.160929    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-kube-api-access-h4sdd" (OuterVolumeSpecName: "kube-api-access-h4sdd") pod "c79ecd78-9893-43a0-81fe-ca9f6cd3149d" (UID: "c79ecd78-9893-43a0-81fe-ca9f6cd3149d"). InnerVolumeSpecName "kube-api-access-h4sdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:07.161201 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.161183    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80a5891-8573-45e1-a741-896aa134f92c-kube-api-access-4drhl" (OuterVolumeSpecName: "kube-api-access-4drhl") pod "d80a5891-8573-45e1-a741-896aa134f92c" (UID: "d80a5891-8573-45e1-a741-896aa134f92c"). InnerVolumeSpecName "kube-api-access-4drhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:07.164594 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.164570    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-util" (OuterVolumeSpecName: "util") pod "c79ecd78-9893-43a0-81fe-ca9f6cd3149d" (UID: "c79ecd78-9893-43a0-81fe-ca9f6cd3149d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.164667 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.164639    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-util" (OuterVolumeSpecName: "util") pod "d80a5891-8573-45e1-a741-896aa134f92c" (UID: "d80a5891-8573-45e1-a741-896aa134f92c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.260219 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260173    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-bundle\") pod \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") "
Jul 02 13:22:07.260326 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260231    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-bundle\") pod \"726977eb-d563-4182-b3d7-bcaea887084a\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") "
Jul 02 13:22:07.260326 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260260    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-util\") pod \"726977eb-d563-4182-b3d7-bcaea887084a\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") "
Jul 02 13:22:07.260326 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260307    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txbd5\" (UniqueName: \"kubernetes.io/projected/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-kube-api-access-txbd5\") pod \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") "
Jul 02 13:22:07.260439 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-util\") pod \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\" (UID: \"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3\") "
Jul 02 13:22:07.260439 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w579v\" (UniqueName: \"kubernetes.io/projected/726977eb-d563-4182-b3d7-bcaea887084a-kube-api-access-w579v\") pod \"726977eb-d563-4182-b3d7-bcaea887084a\" (UID: \"726977eb-d563-4182-b3d7-bcaea887084a\") "
Jul 02 13:22:07.260696 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260674    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.260816 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260699    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4sdd\" (UniqueName: \"kubernetes.io/projected/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-kube-api-access-h4sdd\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.260816 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260716    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.260816 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260730    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c79ecd78-9893-43a0-81fe-ca9f6cd3149d-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.260816 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260742    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d80a5891-8573-45e1-a741-896aa134f92c-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.260816 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260754    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drhl\" (UniqueName: \"kubernetes.io/projected/d80a5891-8573-45e1-a741-896aa134f92c-kube-api-access-4drhl\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.261054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.260893    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-bundle" (OuterVolumeSpecName: "bundle") pod "726977eb-d563-4182-b3d7-bcaea887084a" (UID: "726977eb-d563-4182-b3d7-bcaea887084a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.261124 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.261100    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-bundle" (OuterVolumeSpecName: "bundle") pod "92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" (UID: "92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.262567 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.262547    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-kube-api-access-txbd5" (OuterVolumeSpecName: "kube-api-access-txbd5") pod "92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" (UID: "92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3"). InnerVolumeSpecName "kube-api-access-txbd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:07.262847 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.262828    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726977eb-d563-4182-b3d7-bcaea887084a-kube-api-access-w579v" (OuterVolumeSpecName: "kube-api-access-w579v") pod "726977eb-d563-4182-b3d7-bcaea887084a" (UID: "726977eb-d563-4182-b3d7-bcaea887084a"). InnerVolumeSpecName "kube-api-access-w579v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:22:07.266415 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.266392    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-util" (OuterVolumeSpecName: "util") pod "92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" (UID: "92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.266864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.266844    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-util" (OuterVolumeSpecName: "util") pod "726977eb-d563-4182-b3d7-bcaea887084a" (UID: "726977eb-d563-4182-b3d7-bcaea887084a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:22:07.361253 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.361224    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.361253 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.361249    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-bundle\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.361253 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.361258    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/726977eb-d563-4182-b3d7-bcaea887084a-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.361431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.361266    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbd5\" (UniqueName: \"kubernetes.io/projected/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-kube-api-access-txbd5\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.361431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.361275    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3-util\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.361431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.361283    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w579v\" (UniqueName: \"kubernetes.io/projected/726977eb-d563-4182-b3d7-bcaea887084a-kube-api-access-w579v\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:22:07.840240 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.840217    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj"
Jul 02 13:22:07.840379 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.840207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab5qckj" event={"ID":"c79ecd78-9893-43a0-81fe-ca9f6cd3149d","Type":"ContainerDied","Data":"589917868f0f9354c2b84f73d9d1f431c1c69374c26f77a078aa96f64b0f2e89"}
Jul 02 13:22:07.840379 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.840364    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589917868f0f9354c2b84f73d9d1f431c1c69374c26f77a078aa96f64b0f2e89"
Jul 02 13:22:07.842371 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.842344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr" event={"ID":"d80a5891-8573-45e1-a741-896aa134f92c","Type":"ContainerDied","Data":"8c0771d3ba1542f2691702f4b4f69ec41ed4ac8a42ed91e4e8615a3b8470b481"}
Jul 02 13:22:07.842557 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.842376    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efr4qfr"
Jul 02 13:22:07.842557 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.842390    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0771d3ba1542f2691702f4b4f69ec41ed4ac8a42ed91e4e8615a3b8470b481"
Jul 02 13:22:07.844431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.844403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6" event={"ID":"726977eb-d563-4182-b3d7-bcaea887084a","Type":"ContainerDied","Data":"f6b80726771d10de8e93ba72d7ec17ce7c6cf3c776354c489c74fd766188807a"}
Jul 02 13:22:07.844558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.844447    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b80726771d10de8e93ba72d7ec17ce7c6cf3c776354c489c74fd766188807a"
Jul 02 13:22:07.844558 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.844410    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36sk8q6"
Jul 02 13:22:07.846568 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.846545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6" event={"ID":"92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3","Type":"ContainerDied","Data":"02835cd8661d3fc8442dbb53f854e15ea61b9210ae39e152b15070fd1c394b3d"}
Jul 02 13:22:07.846702 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.846589    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02835cd8661d3fc8442dbb53f854e15ea61b9210ae39e152b15070fd1c394b3d"
Jul 02 13:22:07.846702 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:07.846602    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76zwqp6"
Jul 02 13:22:12.462629 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462591    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"]
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462898    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="util"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462908    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="util"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462918    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="extract"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462923    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="extract"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462932    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="pull"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462939    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="pull"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="extract"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462950    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="extract"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462959    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="extract"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462964    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="extract"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462971    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5fce27-47f9-4546-ab87-2f03ac191068" containerName="console"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462976    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5fce27-47f9-4546-ab87-2f03ac191068" containerName="console"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462984    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="pull"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462989    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="pull"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.462998    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="util"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463003    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="util"
Jul 02 13:22:12.462999 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463009    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="util"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463014    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="util"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463023    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="util"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463028    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="util"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463033    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="pull"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463038    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="pull"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463045    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="extract"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463049    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="extract"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463055    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="pull"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463059    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="pull"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463107    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="726977eb-d563-4182-b3d7-bcaea887084a" containerName="extract"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463115    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5fce27-47f9-4546-ab87-2f03ac191068" containerName="console"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463122    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79ecd78-9893-43a0-81fe-ca9f6cd3149d" containerName="extract"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463130    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b5d3d3-4326-4aa4-9bca-2df6b2d20dc3" containerName="extract"
Jul 02 13:22:12.463474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.463136    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80a5891-8573-45e1-a741-896aa134f92c" containerName="extract"
Jul 02 13:22:12.466208 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.466190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:12.468988 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.468963    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 02 13:22:12.469128 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.469004    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 02 13:22:12.470241 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.470220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7mrn9\""
Jul 02 13:22:12.479561 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.479537    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"]
Jul 02 13:22:12.510248 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.510214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2ph\" (UniqueName: \"kubernetes.io/projected/2b916bad-a834-4461-a5a6-69eba9c73514-kube-api-access-5l2ph\") pod \"limitador-operator-controller-manager-6f9f468797-kfnjg\" (UID: \"2b916bad-a834-4461-a5a6-69eba9c73514\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:12.611354 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.611310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2ph\" (UniqueName: \"kubernetes.io/projected/2b916bad-a834-4461-a5a6-69eba9c73514-kube-api-access-5l2ph\") pod \"limitador-operator-controller-manager-6f9f468797-kfnjg\" (UID: \"2b916bad-a834-4461-a5a6-69eba9c73514\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:12.621759 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.621732    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2ph\" (UniqueName: \"kubernetes.io/projected/2b916bad-a834-4461-a5a6-69eba9c73514-kube-api-access-5l2ph\") pod \"limitador-operator-controller-manager-6f9f468797-kfnjg\" (UID: \"2b916bad-a834-4461-a5a6-69eba9c73514\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:12.776719 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.776636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:12.972234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:12.972206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"]
Jul 02 13:22:12.973695 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:12.973672    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b916bad_a834_4461_a5a6_69eba9c73514.slice/crio-0a437d74a343c3f6c290904982b7d55ac52316bd2ff7a4c182745ff1100448e4 WatchSource:0}: Error finding container 0a437d74a343c3f6c290904982b7d55ac52316bd2ff7a4c182745ff1100448e4: Status 404 returned error can't find the container with id 0a437d74a343c3f6c290904982b7d55ac52316bd2ff7a4c182745ff1100448e4
Jul 02 13:22:13.878128 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:13.878092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg" event={"ID":"2b916bad-a834-4461-a5a6-69eba9c73514","Type":"ContainerStarted","Data":"0a437d74a343c3f6c290904982b7d55ac52316bd2ff7a4c182745ff1100448e4"}
Jul 02 13:22:15.890172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:15.890086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg" event={"ID":"2b916bad-a834-4461-a5a6-69eba9c73514","Type":"ContainerStarted","Data":"b3988da0d05c2bd11a27215309afe4606b75a2ab31670d85a488db9da63a5bf7"}
Jul 02 13:22:15.890521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:15.890210    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:15.908260 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:15.908196    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg" podStartSLOduration=1.250954271 podStartE2EDuration="3.908164773s" totalImagesPullingTime="2.657210502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:15.90782121 +0000 UTC m=+510.638859356" watchObservedRunningTime="2026-07-02 13:22:15.908164773 +0000 UTC m=+510.639202904"
Jul 02 13:22:17.456069 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.456033    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"]
Jul 02 13:22:17.459497 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.459479    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:17.462901 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.462880    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 02 13:22:17.462991 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.462904    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sgn94\""
Jul 02 13:22:17.471146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.471125    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"]
Jul 02 13:22:17.550798 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.550759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2fn\" (UniqueName: \"kubernetes.io/projected/80ee4fa5-5c1f-48f3-8e4a-d1023ba49100-kube-api-access-6c2fn\") pod \"dns-operator-controller-manager-65b49595d7-fkvcz\" (UID: \"80ee4fa5-5c1f-48f3-8e4a-d1023ba49100\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:17.652073 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.652024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2fn\" (UniqueName: \"kubernetes.io/projected/80ee4fa5-5c1f-48f3-8e4a-d1023ba49100-kube-api-access-6c2fn\") pod \"dns-operator-controller-manager-65b49595d7-fkvcz\" (UID: \"80ee4fa5-5c1f-48f3-8e4a-d1023ba49100\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:17.671551 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.671510    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2fn\" (UniqueName: \"kubernetes.io/projected/80ee4fa5-5c1f-48f3-8e4a-d1023ba49100-kube-api-access-6c2fn\") pod \"dns-operator-controller-manager-65b49595d7-fkvcz\" (UID: \"80ee4fa5-5c1f-48f3-8e4a-d1023ba49100\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:17.770250 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.770172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:17.907494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:17.907471    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"]
Jul 02 13:22:17.909093 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:17.909063    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ee4fa5_5c1f_48f3_8e4a_d1023ba49100.slice/crio-b6d0c6bc2627676f988507308bb4ab55a132479f7a6de0d2618501ce36814dcf WatchSource:0}: Error finding container b6d0c6bc2627676f988507308bb4ab55a132479f7a6de0d2618501ce36814dcf: Status 404 returned error can't find the container with id b6d0c6bc2627676f988507308bb4ab55a132479f7a6de0d2618501ce36814dcf
Jul 02 13:22:18.905169 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:18.905133    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz" event={"ID":"80ee4fa5-5c1f-48f3-8e4a-d1023ba49100","Type":"ContainerStarted","Data":"b6d0c6bc2627676f988507308bb4ab55a132479f7a6de0d2618501ce36814dcf"}
Jul 02 13:22:19.911151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:19.911121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz" event={"ID":"80ee4fa5-5c1f-48f3-8e4a-d1023ba49100","Type":"ContainerStarted","Data":"dc860fc2619e048fda7df84d0ac5582cd487dd59a10e889a807ad5acbf2e1a35"}
Jul 02 13:22:19.911510 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:19.911229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:19.943209 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:19.943156    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz" podStartSLOduration=1.064603277 podStartE2EDuration="2.943138746s" totalImagesPullingTime="1.878535469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:19.942323146 +0000 UTC m=+514.673361296" watchObservedRunningTime="2026-07-02 13:22:19.943138746 +0000 UTC m=+514.674176876"
Jul 02 13:22:21.669886 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.669847    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"]
Jul 02 13:22:21.676836 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.676813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.679786 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.679763    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g5bg2\""
Jul 02 13:22:21.689759 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.689724    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"]
Jul 02 13:22:21.690188 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.690157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmcm5\" (UniqueName: \"kubernetes.io/projected/eff42df7-ea1d-460d-bfb6-5892c1cdb86e-kube-api-access-wmcm5\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9\" (UID: \"eff42df7-ea1d-460d-bfb6-5892c1cdb86e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.690299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.690239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff42df7-ea1d-460d-bfb6-5892c1cdb86e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9\" (UID: \"eff42df7-ea1d-460d-bfb6-5892c1cdb86e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.790824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.790789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff42df7-ea1d-460d-bfb6-5892c1cdb86e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9\" (UID: \"eff42df7-ea1d-460d-bfb6-5892c1cdb86e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.790988 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.790870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcm5\" (UniqueName: \"kubernetes.io/projected/eff42df7-ea1d-460d-bfb6-5892c1cdb86e-kube-api-access-wmcm5\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9\" (UID: \"eff42df7-ea1d-460d-bfb6-5892c1cdb86e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.791207 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.791184    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff42df7-ea1d-460d-bfb6-5892c1cdb86e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9\" (UID: \"eff42df7-ea1d-460d-bfb6-5892c1cdb86e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.800393 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.800363    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcm5\" (UniqueName: \"kubernetes.io/projected/eff42df7-ea1d-460d-bfb6-5892c1cdb86e-kube-api-access-wmcm5\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9\" (UID: \"eff42df7-ea1d-460d-bfb6-5892c1cdb86e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:21.990666 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:21.990562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:22.133602 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:22.133574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"]
Jul 02 13:22:22.135354 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:22:22.135321    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff42df7_ea1d_460d_bfb6_5892c1cdb86e.slice/crio-e19577ab03f23ea047641423ccd33c811b60db58337031b6fe3c65d5cbc5618d WatchSource:0}: Error finding container e19577ab03f23ea047641423ccd33c811b60db58337031b6fe3c65d5cbc5618d: Status 404 returned error can't find the container with id e19577ab03f23ea047641423ccd33c811b60db58337031b6fe3c65d5cbc5618d
Jul 02 13:22:22.927756 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:22.927722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9" event={"ID":"eff42df7-ea1d-460d-bfb6-5892c1cdb86e","Type":"ContainerStarted","Data":"e19577ab03f23ea047641423ccd33c811b60db58337031b6fe3c65d5cbc5618d"}
Jul 02 13:22:26.897573 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:26.897541    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg"
Jul 02 13:22:27.957928 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:27.957880    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9" event={"ID":"eff42df7-ea1d-460d-bfb6-5892c1cdb86e","Type":"ContainerStarted","Data":"b087aef5581f3b5af5c98601f986deecd5c786ef5571651d53c3440b12063230"}
Jul 02 13:22:27.958297 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:27.958083    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:22:27.988006 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:27.987955    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9" podStartSLOduration=1.767035801 podStartE2EDuration="6.987938995s" totalImagesPullingTime="5.220903194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:27.985078459 +0000 UTC m=+522.716116601" watchObservedRunningTime="2026-07-02 13:22:27.987938995 +0000 UTC m=+522.718977136"
Jul 02 13:22:30.919682 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:30.919650    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz"
Jul 02 13:22:38.965457 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:22:38.965373    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9"
Jul 02 13:23:45.737745 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.737711    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:23:45.739305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.739282    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:23:45.827566 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.827517    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-w6jnj"]
Jul 02 13:23:45.827771 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.827730    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj" podUID="8e30eb35-606a-45ba-912d-7929ad10b3f3" containerName="manager" containerID="cri-o://df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d" gracePeriod=10
Jul 02 13:23:45.854440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.854406    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-pmlpc"]
Jul 02 13:23:45.857630 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.857610    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:45.867894 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:45.867871    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-pmlpc"]
Jul 02 13:23:46.000062 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.000027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f8a0bd-838c-4389-b512-2a05e88d0803-cert\") pod \"kserve-controller-manager-694f5b7594-pmlpc\" (UID: \"d0f8a0bd-838c-4389-b512-2a05e88d0803\") " pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.000192 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.000159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fww69\" (UniqueName: \"kubernetes.io/projected/d0f8a0bd-838c-4389-b512-2a05e88d0803-kube-api-access-fww69\") pod \"kserve-controller-manager-694f5b7594-pmlpc\" (UID: \"d0f8a0bd-838c-4389-b512-2a05e88d0803\") " pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.085697 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.085670    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:23:46.100917 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.100892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f8a0bd-838c-4389-b512-2a05e88d0803-cert\") pod \"kserve-controller-manager-694f5b7594-pmlpc\" (UID: \"d0f8a0bd-838c-4389-b512-2a05e88d0803\") " pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.101014 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.100969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fww69\" (UniqueName: \"kubernetes.io/projected/d0f8a0bd-838c-4389-b512-2a05e88d0803-kube-api-access-fww69\") pod \"kserve-controller-manager-694f5b7594-pmlpc\" (UID: \"d0f8a0bd-838c-4389-b512-2a05e88d0803\") " pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.103142 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.103114    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f8a0bd-838c-4389-b512-2a05e88d0803-cert\") pod \"kserve-controller-manager-694f5b7594-pmlpc\" (UID: \"d0f8a0bd-838c-4389-b512-2a05e88d0803\") " pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.111653 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.111621    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fww69\" (UniqueName: \"kubernetes.io/projected/d0f8a0bd-838c-4389-b512-2a05e88d0803-kube-api-access-fww69\") pod \"kserve-controller-manager-694f5b7594-pmlpc\" (UID: \"d0f8a0bd-838c-4389-b512-2a05e88d0803\") " pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.201725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.201696    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e30eb35-606a-45ba-912d-7929ad10b3f3-cert\") pod \"8e30eb35-606a-45ba-912d-7929ad10b3f3\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") "
Jul 02 13:23:46.201921 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.201749    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pl8d\" (UniqueName: \"kubernetes.io/projected/8e30eb35-606a-45ba-912d-7929ad10b3f3-kube-api-access-7pl8d\") pod \"8e30eb35-606a-45ba-912d-7929ad10b3f3\" (UID: \"8e30eb35-606a-45ba-912d-7929ad10b3f3\") "
Jul 02 13:23:46.203838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.203810    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e30eb35-606a-45ba-912d-7929ad10b3f3-cert" (OuterVolumeSpecName: "cert") pod "8e30eb35-606a-45ba-912d-7929ad10b3f3" (UID: "8e30eb35-606a-45ba-912d-7929ad10b3f3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:23:46.203924 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.203875    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e30eb35-606a-45ba-912d-7929ad10b3f3-kube-api-access-7pl8d" (OuterVolumeSpecName: "kube-api-access-7pl8d") pod "8e30eb35-606a-45ba-912d-7929ad10b3f3" (UID: "8e30eb35-606a-45ba-912d-7929ad10b3f3"). InnerVolumeSpecName "kube-api-access-7pl8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:46.204964 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.204949    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:46.303074 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.303038    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e30eb35-606a-45ba-912d-7929ad10b3f3-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:23:46.303074 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.303069    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pl8d\" (UniqueName: \"kubernetes.io/projected/8e30eb35-606a-45ba-912d-7929ad10b3f3-kube-api-access-7pl8d\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:23:46.339278 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.339252    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-pmlpc"]
Jul 02 13:23:46.341540 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:23:46.341502    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f8a0bd_838c_4389_b512_2a05e88d0803.slice/crio-ae92650ff7067c3d0c2af65ebe29f510029a9c10c8a9af10d401f4cc1db1abeb WatchSource:0}: Error finding container ae92650ff7067c3d0c2af65ebe29f510029a9c10c8a9af10d401f4cc1db1abeb: Status 404 returned error can't find the container with id ae92650ff7067c3d0c2af65ebe29f510029a9c10c8a9af10d401f4cc1db1abeb
Jul 02 13:23:46.350925 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.350901    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e30eb35-606a-45ba-912d-7929ad10b3f3" containerID="df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d" exitCode=0
Jul 02 13:23:46.351020 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.350944    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj"
Jul 02 13:23:46.351020 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.350979    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj" event={"ID":"8e30eb35-606a-45ba-912d-7929ad10b3f3","Type":"ContainerDied","Data":"df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d"}
Jul 02 13:23:46.351118 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.351028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-w6jnj" event={"ID":"8e30eb35-606a-45ba-912d-7929ad10b3f3","Type":"ContainerDied","Data":"f73e959eb789e07be29377c25fdaadbe86b0ae35e89ebbfd667a0a555fce492a"}
Jul 02 13:23:46.351118 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.351055    2564 scope.go:117] "RemoveContainer" containerID="df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d"
Jul 02 13:23:46.352324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.352303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc" event={"ID":"d0f8a0bd-838c-4389-b512-2a05e88d0803","Type":"ContainerStarted","Data":"ae92650ff7067c3d0c2af65ebe29f510029a9c10c8a9af10d401f4cc1db1abeb"}
Jul 02 13:23:46.359997 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.359980    2564 scope.go:117] "RemoveContainer" containerID="df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d"
Jul 02 13:23:46.360286 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:23:46.360252    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d\": container with ID starting with df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d not found: ID does not exist" containerID="df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d"
Jul 02 13:23:46.360390 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.360293    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d"} err="failed to get container status \"df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d\": rpc error: code = NotFound desc = could not find container \"df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d\": container with ID starting with df511c2b83eabf134ed70a498219fae50c5e63d520676d17db7b0d89bcb4628d not found: ID does not exist"
Jul 02 13:23:46.377685 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.377661    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-w6jnj"]
Jul 02 13:23:46.383857 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:46.383834    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-w6jnj"]
Jul 02 13:23:47.359299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:47.359260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc" event={"ID":"d0f8a0bd-838c-4389-b512-2a05e88d0803","Type":"ContainerStarted","Data":"51682046290fb026daf1df7fee73b800a3817dbaddcfd35d1e6b983efbde6832"}
Jul 02 13:23:47.359683 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:47.359365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:23:47.378260 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:47.378182    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc" podStartSLOduration=1.966967839 podStartE2EDuration="2.378165954s" totalImagesPullingTime="411.198115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:47.377560448 +0000 UTC m=+602.108598598" watchObservedRunningTime="2026-07-02 13:23:47.378165954 +0000 UTC m=+602.109204083"
Jul 02 13:23:47.759899 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:23:47.759867    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e30eb35-606a-45ba-912d-7929ad10b3f3" path="/var/lib/kubelet/pods/8e30eb35-606a-45ba-912d-7929ad10b3f3/volumes"
Jul 02 13:24:18.368168 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:18.368089    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-pmlpc"
Jul 02 13:24:19.368780 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.368728    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zzskt"]
Jul 02 13:24:19.369255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.369227    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e30eb35-606a-45ba-912d-7929ad10b3f3" containerName="manager"
Jul 02 13:24:19.369255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.369247    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e30eb35-606a-45ba-912d-7929ad10b3f3" containerName="manager"
Jul 02 13:24:19.369379 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.369364    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e30eb35-606a-45ba-912d-7929ad10b3f3" containerName="manager"
Jul 02 13:24:19.372143 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.372120    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:19.375673 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.375648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 13:24:19.375832 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.375724    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7fhj8\""
Jul 02 13:24:19.383928 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.383904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zzskt"]
Jul 02 13:24:19.471619 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.471582    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd94d58-1c1c-45f6-a911-1b879fc05b86-cert\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:19.471770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.471625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfh6\" (UniqueName: \"kubernetes.io/projected/9dd94d58-1c1c-45f6-a911-1b879fc05b86-kube-api-access-rbfh6\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:19.572956 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.572921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd94d58-1c1c-45f6-a911-1b879fc05b86-cert\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:19.573131 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.572966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfh6\" (UniqueName: \"kubernetes.io/projected/9dd94d58-1c1c-45f6-a911-1b879fc05b86-kube-api-access-rbfh6\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:19.573131 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:24:19.573080    2564 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 02 13:24:19.573271 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:24:19.573165    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd94d58-1c1c-45f6-a911-1b879fc05b86-cert podName:9dd94d58-1c1c-45f6-a911-1b879fc05b86 nodeName:}" failed. No retries permitted until 2026-07-02 13:24:20.073144715 +0000 UTC m=+634.804182840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9dd94d58-1c1c-45f6-a911-1b879fc05b86-cert") pod "odh-model-controller-84b54b7f97-zzskt" (UID: "9dd94d58-1c1c-45f6-a911-1b879fc05b86") : secret "odh-model-controller-webhook-cert" not found
Jul 02 13:24:19.582466 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:19.582438    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfh6\" (UniqueName: \"kubernetes.io/projected/9dd94d58-1c1c-45f6-a911-1b879fc05b86-kube-api-access-rbfh6\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:20.076482 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:20.076446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd94d58-1c1c-45f6-a911-1b879fc05b86-cert\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:20.078897 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:20.078866    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd94d58-1c1c-45f6-a911-1b879fc05b86-cert\") pod \"odh-model-controller-84b54b7f97-zzskt\" (UID: \"9dd94d58-1c1c-45f6-a911-1b879fc05b86\") " pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:20.282868 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:20.282826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:20.432042 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:20.432009    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zzskt"]
Jul 02 13:24:20.433943 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:24:20.433917    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd94d58_1c1c_45f6_a911_1b879fc05b86.slice/crio-69e9623680101f75dda0310b63b56b4dc63cc55162df929fa3002d8fbc2c77c4 WatchSource:0}: Error finding container 69e9623680101f75dda0310b63b56b4dc63cc55162df929fa3002d8fbc2c77c4: Status 404 returned error can't find the container with id 69e9623680101f75dda0310b63b56b4dc63cc55162df929fa3002d8fbc2c77c4
Jul 02 13:24:20.435253 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:20.435233    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:24:20.524782 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:20.524735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zzskt" event={"ID":"9dd94d58-1c1c-45f6-a911-1b879fc05b86","Type":"ContainerStarted","Data":"69e9623680101f75dda0310b63b56b4dc63cc55162df929fa3002d8fbc2c77c4"}
Jul 02 13:24:24.551459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:24.551409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zzskt" event={"ID":"9dd94d58-1c1c-45f6-a911-1b879fc05b86","Type":"ContainerStarted","Data":"70d9eec64b032eed407d1febe90233a5f4cc5dc5b60ccf5b81d84b42a2938359"}
Jul 02 13:24:24.552099 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:24.551551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:24:24.573345 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:24.573298    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-zzskt" podStartSLOduration=2.316547438 podStartE2EDuration="5.573280558s" totalImagesPullingTime="3.25673312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:24.57078952 +0000 UTC m=+639.301827700" watchObservedRunningTime="2026-07-02 13:24:24.573280558 +0000 UTC m=+639.304318689"
Jul 02 13:24:35.560147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:24:35.560115    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-zzskt"
Jul 02 13:26:31.879051 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.879014    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"]
Jul 02 13:26:31.882807 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.882783    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:31.890256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.889553    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:26:31.890256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.889716    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 02 13:26:31.890256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.889775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:26:31.890256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.890053    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:26:31.896644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.896621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"]
Jul 02 13:26:31.991234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.991197    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:31.991443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.991264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:31.991443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.991309    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:31.991443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.991342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99e193-25d7-413f-9158-ec8c33b681f6-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:31.991443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.991384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lfk\" (UniqueName: \"kubernetes.io/projected/ec99e193-25d7-413f-9158-ec8c33b681f6-kube-api-access-s9lfk\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:31.991443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:31.991430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.092583 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.092540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99e193-25d7-413f-9158-ec8c33b681f6-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.092769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.092598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lfk\" (UniqueName: \"kubernetes.io/projected/ec99e193-25d7-413f-9158-ec8c33b681f6-kube-api-access-s9lfk\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.092769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.092639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.092769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.092679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.092769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.092738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.092769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.092765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.093164 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.093144    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.093288 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.093254    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.093422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.093260    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.095036 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.095017    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.095290 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.095270    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99e193-25d7-413f-9158-ec8c33b681f6-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.103887 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.103859    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lfk\" (UniqueName: \"kubernetes.io/projected/ec99e193-25d7-413f-9158-ec8c33b681f6-kube-api-access-s9lfk\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.195740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.195382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:32.339278 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:32.339235    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"]
Jul 02 13:26:32.340992 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:26:32.340963    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec99e193_25d7_413f_9158_ec8c33b681f6.slice/crio-29f78a4dc9d27447f7a0efacfe796079ef8b3832ae5aaf0bf018f5b8450b7da2 WatchSource:0}: Error finding container 29f78a4dc9d27447f7a0efacfe796079ef8b3832ae5aaf0bf018f5b8450b7da2: Status 404 returned error can't find the container with id 29f78a4dc9d27447f7a0efacfe796079ef8b3832ae5aaf0bf018f5b8450b7da2
Jul 02 13:26:33.207421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:33.207375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq" event={"ID":"ec99e193-25d7-413f-9158-ec8c33b681f6","Type":"ContainerStarted","Data":"29f78a4dc9d27447f7a0efacfe796079ef8b3832ae5aaf0bf018f5b8450b7da2"}
Jul 02 13:26:34.215067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:34.215029    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq" event={"ID":"ec99e193-25d7-413f-9158-ec8c33b681f6","Type":"ContainerStarted","Data":"c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2"}
Jul 02 13:26:34.259579 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:34.259498    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq" podStartSLOduration=2.196002182 podStartE2EDuration="3.259481291s" totalImagesPullingTime="1.063479109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:34.257843024 +0000 UTC m=+768.988881167" watchObservedRunningTime="2026-07-02 13:26:34.259481291 +0000 UTC m=+768.990519421"
Jul 02 13:26:35.197326 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:35.197275    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:42.196410 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.196363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:42.209313 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.209288    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:42.267253 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.267227    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:26:42.468316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.468232    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"]
Jul 02 13:26:42.471680 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.471664    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.480404 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.480379    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:26:42.480546 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.480430    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 02 13:26:42.507895 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.507867    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"]
Jul 02 13:26:42.587761 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-home\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.587761 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4h4\" (UniqueName: \"kubernetes.io/projected/cd565435-bf6e-43df-b936-f3619e47476a-kube-api-access-ph4h4\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.587982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-model-cache\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.587982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd565435-bf6e-43df-b936-f3619e47476a-tls-certs\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.587982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-dshm\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.587982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.587982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.587962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.688823 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.688792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd565435-bf6e-43df-b936-f3619e47476a-tls-certs\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.688994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.688827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-dshm\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.688994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.688850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.688994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.688977    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-home\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4h4\" (UniqueName: \"kubernetes.io/projected/cd565435-bf6e-43df-b936-f3619e47476a-kube-api-access-ph4h4\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689285 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-model-cache\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689285 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689209    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689426 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689334    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689477 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689451    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-model-cache\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.689565 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.689518    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-home\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.691769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.691736    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-dshm\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.691980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.691964    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd565435-bf6e-43df-b936-f3619e47476a-tls-certs\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.699249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.699225    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4h4\" (UniqueName: \"kubernetes.io/projected/cd565435-bf6e-43df-b936-f3619e47476a-kube-api-access-ph4h4\") pod \"precise-prefix-cache-test-kserve-5767855db4-ggdxw\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.781974 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.781900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:42.938218 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:42.938193    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"]
Jul 02 13:26:42.940301 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:26:42.940267    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd565435_bf6e_43df_b936_f3619e47476a.slice/crio-c4c086fb46e0bad2c770c7f28205ff59b1f327d896f5beb9ee910893a8b718cc WatchSource:0}: Error finding container c4c086fb46e0bad2c770c7f28205ff59b1f327d896f5beb9ee910893a8b718cc: Status 404 returned error can't find the container with id c4c086fb46e0bad2c770c7f28205ff59b1f327d896f5beb9ee910893a8b718cc
Jul 02 13:26:43.263032 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:43.262986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" event={"ID":"cd565435-bf6e-43df-b936-f3619e47476a","Type":"ContainerStarted","Data":"c4c086fb46e0bad2c770c7f28205ff59b1f327d896f5beb9ee910893a8b718cc"}
Jul 02 13:26:45.791413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:45.791376    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"]
Jul 02 13:26:46.477082 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:46.477056    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:26:47.295286 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:47.295247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" event={"ID":"cd565435-bf6e-43df-b936-f3619e47476a","Type":"ContainerStarted","Data":"efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293"}
Jul 02 13:26:51.319073 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:51.319039    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd565435-bf6e-43df-b936-f3619e47476a" containerID="efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293" exitCode=0
Jul 02 13:26:51.319459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:51.319092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" event={"ID":"cd565435-bf6e-43df-b936-f3619e47476a","Type":"ContainerDied","Data":"efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293"}
Jul 02 13:26:52.326363 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:52.326317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" event={"ID":"cd565435-bf6e-43df-b936-f3619e47476a","Type":"ContainerStarted","Data":"b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41"}
Jul 02 13:26:52.348720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:52.348671    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" podStartSLOduration=2.76427063 podStartE2EDuration="10.348650505s" totalImagesPullingTime="3.531672976s" totalInitContainerRuntime="4.052706899s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:52.347370978 +0000 UTC m=+787.078409123" watchObservedRunningTime="2026-07-02 13:26:52.348650505 +0000 UTC m=+787.079688635"
Jul 02 13:26:52.782441 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:52.782407    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:52.782626 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:52.782483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:52.795541 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:52.795490    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:26:53.343820 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:26:53.343783    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:27:00.815821 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:00.815753    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq" podUID="ec99e193-25d7-413f-9158-ec8c33b681f6" containerName="main" containerID="cri-o://c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2" gracePeriod=45
Jul 02 13:27:01.121175 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.121149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:27:01.157001 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.156962    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-dshm\") pod \"ec99e193-25d7-413f-9158-ec8c33b681f6\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") "
Jul 02 13:27:01.157183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157008    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-home\") pod \"ec99e193-25d7-413f-9158-ec8c33b681f6\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") "
Jul 02 13:27:01.157183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157037    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-tmp-dir\") pod \"ec99e193-25d7-413f-9158-ec8c33b681f6\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") "
Jul 02 13:27:01.157183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-model-cache\") pod \"ec99e193-25d7-413f-9158-ec8c33b681f6\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") "
Jul 02 13:27:01.157183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99e193-25d7-413f-9158-ec8c33b681f6-tls-certs\") pod \"ec99e193-25d7-413f-9158-ec8c33b681f6\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") "
Jul 02 13:27:01.157183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157181    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9lfk\" (UniqueName: \"kubernetes.io/projected/ec99e193-25d7-413f-9158-ec8c33b681f6-kube-api-access-s9lfk\") pod \"ec99e193-25d7-413f-9158-ec8c33b681f6\" (UID: \"ec99e193-25d7-413f-9158-ec8c33b681f6\") "
Jul 02 13:27:01.157461 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157337    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-home" (OuterVolumeSpecName: "home") pod "ec99e193-25d7-413f-9158-ec8c33b681f6" (UID: "ec99e193-25d7-413f-9158-ec8c33b681f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:01.157461 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157351    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec99e193-25d7-413f-9158-ec8c33b681f6" (UID: "ec99e193-25d7-413f-9158-ec8c33b681f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:01.157461 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157358    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec99e193-25d7-413f-9158-ec8c33b681f6" (UID: "ec99e193-25d7-413f-9158-ec8c33b681f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:01.157654 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157465    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:27:01.157654 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157490    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:27:01.157654 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.157506    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:27:01.159399 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.159374    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-dshm" (OuterVolumeSpecName: "dshm") pod "ec99e193-25d7-413f-9158-ec8c33b681f6" (UID: "ec99e193-25d7-413f-9158-ec8c33b681f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:01.159543 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.159426    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec99e193-25d7-413f-9158-ec8c33b681f6-kube-api-access-s9lfk" (OuterVolumeSpecName: "kube-api-access-s9lfk") pod "ec99e193-25d7-413f-9158-ec8c33b681f6" (UID: "ec99e193-25d7-413f-9158-ec8c33b681f6"). InnerVolumeSpecName "kube-api-access-s9lfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:27:01.159543 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.159472    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec99e193-25d7-413f-9158-ec8c33b681f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec99e193-25d7-413f-9158-ec8c33b681f6" (UID: "ec99e193-25d7-413f-9158-ec8c33b681f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:27:01.258494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.258452    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec99e193-25d7-413f-9158-ec8c33b681f6-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:27:01.258494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.258484    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec99e193-25d7-413f-9158-ec8c33b681f6-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:27:01.258494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.258499    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lfk\" (UniqueName: \"kubernetes.io/projected/ec99e193-25d7-413f-9158-ec8c33b681f6-kube-api-access-s9lfk\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:27:01.376585 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.376552    2564 generic.go:356] "Generic (PLEG): container finished" podID="ec99e193-25d7-413f-9158-ec8c33b681f6" containerID="c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2" exitCode=0
Jul 02 13:27:01.376742 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.376605    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"
Jul 02 13:27:01.376742 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.376639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq" event={"ID":"ec99e193-25d7-413f-9158-ec8c33b681f6","Type":"ContainerDied","Data":"c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2"}
Jul 02 13:27:01.376742 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.376694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq" event={"ID":"ec99e193-25d7-413f-9158-ec8c33b681f6","Type":"ContainerDied","Data":"29f78a4dc9d27447f7a0efacfe796079ef8b3832ae5aaf0bf018f5b8450b7da2"}
Jul 02 13:27:01.376742 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.376719    2564 scope.go:117] "RemoveContainer" containerID="c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2"
Jul 02 13:27:01.386358 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.386328    2564 scope.go:117] "RemoveContainer" containerID="c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2"
Jul 02 13:27:01.386665 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:27:01.386646    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2\": container with ID starting with c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2 not found: ID does not exist" containerID="c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2"
Jul 02 13:27:01.386746 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.386676    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2"} err="failed to get container status \"c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2\": rpc error: code = NotFound desc = could not find container \"c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2\": container with ID starting with c46db39b90691cbb6fbbb118eb9a889e2054861cd03fc80261709dccb7f838e2 not found: ID does not exist"
Jul 02 13:27:01.399949 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.399924    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"]
Jul 02 13:27:01.407191 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.407170    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq"]
Jul 02 13:27:01.760971 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:01.760893    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec99e193-25d7-413f-9158-ec8c33b681f6" path="/var/lib/kubelet/pods/ec99e193-25d7-413f-9158-ec8c33b681f6/volumes"
Jul 02 13:27:23.424978 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.424890    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"]
Jul 02 13:27:23.425487 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.425467    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec99e193-25d7-413f-9158-ec8c33b681f6" containerName="main"
Jul 02 13:27:23.425608 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.425490    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec99e193-25d7-413f-9158-ec8c33b681f6" containerName="main"
Jul 02 13:27:23.425608 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.425603    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec99e193-25d7-413f-9158-ec8c33b681f6" containerName="main"
Jul 02 13:27:23.428934 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.428911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.433258 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.433238    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 02 13:27:23.447710 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.447678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"]
Jul 02 13:27:23.550124 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hz6\" (UniqueName: \"kubernetes.io/projected/10c8a29f-0e04-498c-ab4a-e2a910726d24-kube-api-access-j4hz6\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.550281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-model-cache\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.550281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-home\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.550281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-dshm\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.550281 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550226    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c8a29f-0e04-498c-ab4a-e2a910726d24-tls-certs\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.550456 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-tmp-dir\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.550456 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.550337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-kserve-provision-location\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.651644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hz6\" (UniqueName: \"kubernetes.io/projected/10c8a29f-0e04-498c-ab4a-e2a910726d24-kube-api-access-j4hz6\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.651826 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-model-cache\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.651826 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-home\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.651826 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-dshm\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.651826 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651750    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c8a29f-0e04-498c-ab4a-e2a910726d24-tls-certs\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.651826 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-tmp-dir\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.652042 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.651833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-kserve-provision-location\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.652168 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.652144    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-home\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.652232 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.652182    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-model-cache\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.652289 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.652274    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-tmp-dir\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.652359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.652337    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-kserve-provision-location\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.654084 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.654061    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-dshm\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.654276 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.654256    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c8a29f-0e04-498c-ab4a-e2a910726d24-tls-certs\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.666010 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.665980    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hz6\" (UniqueName: \"kubernetes.io/projected/10c8a29f-0e04-498c-ab4a-e2a910726d24-kube-api-access-j4hz6\") pod \"auth-enabled-test-kserve-dbcffbc7b-pf95g\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.683692 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.683621    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"]
Jul 02 13:27:23.687680 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.687660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.692779 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.692752    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-2nz5n\""
Jul 02 13:27:23.708030 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.708005    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"]
Jul 02 13:27:23.739462 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.739428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:27:23.753017 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.752988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.753161 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.753081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.753227 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.753181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvtm\" (UniqueName: \"kubernetes.io/projected/3e96e63f-0643-437c-8321-294005b83acd-kube-api-access-7hvtm\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.753283 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.753242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e96e63f-0643-437c-8321-294005b83acd-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.854163 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.854124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.854371 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.854240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvtm\" (UniqueName: \"kubernetes.io/projected/3e96e63f-0643-437c-8321-294005b83acd-kube-api-access-7hvtm\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.854371 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.854288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e96e63f-0643-437c-8321-294005b83acd-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.854371 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.854319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.854741 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.854650    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.854741 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.854703    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.857168 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.857142    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e96e63f-0643-437c-8321-294005b83acd-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.866948 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.866914    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvtm\" (UniqueName: \"kubernetes.io/projected/3e96e63f-0643-437c-8321-294005b83acd-kube-api-access-7hvtm\") pod \"auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:23.913636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.913608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"]
Jul 02 13:27:23.915635 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:27:23.915601    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c8a29f_0e04_498c_ab4a_e2a910726d24.slice/crio-3e3f93a5a1c906d6b2f8309ed0fca2a1ac83b1ddfaf5b29ecb367301e4748d16 WatchSource:0}: Error finding container 3e3f93a5a1c906d6b2f8309ed0fca2a1ac83b1ddfaf5b29ecb367301e4748d16: Status 404 returned error can't find the container with id 3e3f93a5a1c906d6b2f8309ed0fca2a1ac83b1ddfaf5b29ecb367301e4748d16
Jul 02 13:27:23.997694 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:23.997659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:24.160662 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:24.160636    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"]
Jul 02 13:27:24.163823 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:27:24.163790    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e96e63f_0643_437c_8321_294005b83acd.slice/crio-35f91947f1195969f17a7a04dd8e9050201e0831954127e1a69ed5f5f27fcfe9 WatchSource:0}: Error finding container 35f91947f1195969f17a7a04dd8e9050201e0831954127e1a69ed5f5f27fcfe9: Status 404 returned error can't find the container with id 35f91947f1195969f17a7a04dd8e9050201e0831954127e1a69ed5f5f27fcfe9
Jul 02 13:27:24.504798 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:24.504579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" event={"ID":"10c8a29f-0e04-498c-ab4a-e2a910726d24","Type":"ContainerStarted","Data":"09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59"}
Jul 02 13:27:24.504798 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:24.504708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" event={"ID":"10c8a29f-0e04-498c-ab4a-e2a910726d24","Type":"ContainerStarted","Data":"3e3f93a5a1c906d6b2f8309ed0fca2a1ac83b1ddfaf5b29ecb367301e4748d16"}
Jul 02 13:27:24.507087 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:24.507056    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" event={"ID":"3e96e63f-0643-437c-8321-294005b83acd","Type":"ContainerStarted","Data":"8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6"}
Jul 02 13:27:24.507222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:24.507103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" event={"ID":"3e96e63f-0643-437c-8321-294005b83acd","Type":"ContainerStarted","Data":"35f91947f1195969f17a7a04dd8e9050201e0831954127e1a69ed5f5f27fcfe9"}
Jul 02 13:27:25.513817 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:25.513782    2564 generic.go:356] "Generic (PLEG): container finished" podID="3e96e63f-0643-437c-8321-294005b83acd" containerID="8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6" exitCode=0
Jul 02 13:27:25.514221 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:25.513860    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" event={"ID":"3e96e63f-0643-437c-8321-294005b83acd","Type":"ContainerDied","Data":"8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6"}
Jul 02 13:27:27.527572 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:27.527510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" event={"ID":"3e96e63f-0643-437c-8321-294005b83acd","Type":"ContainerStarted","Data":"b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"}
Jul 02 13:27:27.527977 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:27.527656    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:27:27.555177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:27.555124    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" podStartSLOduration=2.524817632 podStartE2EDuration="4.555108053s" totalImagesPullingTime="1.265275659s" totalInitContainerRuntime="765.014762ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:27:27.553965242 +0000 UTC m=+822.285003395" watchObservedRunningTime="2026-07-02 13:27:27.555108053 +0000 UTC m=+822.286146183"
Jul 02 13:27:27.767612 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:27:27.767576    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c8a29f_0e04_498c_ab4a_e2a910726d24.slice/crio-09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59.scope\": RecentStats: unable to find data in memory cache]"
Jul 02 13:27:28.535279 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:28.535246    2564 generic.go:356] "Generic (PLEG): container finished" podID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerID="09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59" exitCode=0
Jul 02 13:27:28.535719 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:28.535320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" event={"ID":"10c8a29f-0e04-498c-ab4a-e2a910726d24","Type":"ContainerDied","Data":"09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59"}
Jul 02 13:27:55.502324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:55.502282    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"]
Jul 02 13:27:58.538240 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:27:58.538207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:28:10.531599 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:10.531503    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" podUID="cd565435-bf6e-43df-b936-f3619e47476a" containerName="main" containerID="cri-o://b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41" gracePeriod=45
Jul 02 13:28:11.220191 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.220164    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:28:11.308247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph4h4\" (UniqueName: \"kubernetes.io/projected/cd565435-bf6e-43df-b936-f3619e47476a-kube-api-access-ph4h4\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-model-cache\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308386 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308290    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-kserve-provision-location\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308386 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-home\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308386 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308345    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-tmp-dir\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308552 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308389    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd565435-bf6e-43df-b936-f3619e47476a-tls-certs\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308552 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308418    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-dshm\") pod \"cd565435-bf6e-43df-b936-f3619e47476a\" (UID: \"cd565435-bf6e-43df-b936-f3619e47476a\") "
Jul 02 13:28:11.308649 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308572    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:11.308704 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308650    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:11.308704 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308687    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.308812 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.308714    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-home" (OuterVolumeSpecName: "home") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:11.310396 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.310363    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd565435-bf6e-43df-b936-f3619e47476a-kube-api-access-ph4h4" (OuterVolumeSpecName: "kube-api-access-ph4h4") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "kube-api-access-ph4h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:28:11.310492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.310451    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd565435-bf6e-43df-b936-f3619e47476a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:28:11.310588 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.310557    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-dshm" (OuterVolumeSpecName: "dshm") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:11.366789 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.366752    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd565435-bf6e-43df-b936-f3619e47476a" (UID: "cd565435-bf6e-43df-b936-f3619e47476a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:11.409717 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.409693    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.409717 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.409716    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.409867 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.409725    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd565435-bf6e-43df-b936-f3619e47476a-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.409867 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.409733    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.409867 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.409742    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph4h4\" (UniqueName: \"kubernetes.io/projected/cd565435-bf6e-43df-b936-f3619e47476a-kube-api-access-ph4h4\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.409867 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.409750    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd565435-bf6e-43df-b936-f3619e47476a-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.800774 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.800734    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd565435-bf6e-43df-b936-f3619e47476a" containerID="b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41" exitCode=0
Jul 02 13:28:11.801213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.800788    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"
Jul 02 13:28:11.801213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.800824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" event={"ID":"cd565435-bf6e-43df-b936-f3619e47476a","Type":"ContainerDied","Data":"b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41"}
Jul 02 13:28:11.801213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.800875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw" event={"ID":"cd565435-bf6e-43df-b936-f3619e47476a","Type":"ContainerDied","Data":"c4c086fb46e0bad2c770c7f28205ff59b1f327d896f5beb9ee910893a8b718cc"}
Jul 02 13:28:11.801213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.800902    2564 scope.go:117] "RemoveContainer" containerID="b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41"
Jul 02 13:28:11.803720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.803694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" event={"ID":"10c8a29f-0e04-498c-ab4a-e2a910726d24","Type":"ContainerStarted","Data":"d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395"}
Jul 02 13:28:11.810048 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.810030    2564 scope.go:117] "RemoveContainer" containerID="efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293"
Jul 02 13:28:11.820735 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.820716    2564 scope.go:117] "RemoveContainer" containerID="b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41"
Jul 02 13:28:11.821007 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:28:11.820986    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41\": container with ID starting with b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41 not found: ID does not exist" containerID="b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41"
Jul 02 13:28:11.821090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.821020    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41"} err="failed to get container status \"b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41\": rpc error: code = NotFound desc = could not find container \"b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41\": container with ID starting with b91ca556ac76844d8414df56008170a5bbc1f4ce60e241b3e766495ef8804a41 not found: ID does not exist"
Jul 02 13:28:11.821090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.821043    2564 scope.go:117] "RemoveContainer" containerID="efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293"
Jul 02 13:28:11.821270 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:28:11.821251    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293\": container with ID starting with efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293 not found: ID does not exist" containerID="efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293"
Jul 02 13:28:11.821312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.821280    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293"} err="failed to get container status \"efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293\": rpc error: code = NotFound desc = could not find container \"efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293\": container with ID starting with efac4ae65f9ae0c4e5dc08295d09191fdac08dcc647f13477ac6cad4f15cf293 not found: ID does not exist"
Jul 02 13:28:11.829260 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.829238    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"]
Jul 02 13:28:11.843703 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.843665    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw"]
Jul 02 13:28:11.876223 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:11.876176    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podStartSLOduration=2.343963863 podStartE2EDuration="48.876160726s" totalImagesPullingTime="42.744524415s" totalInitContainerRuntime="3.787672448s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:28:11.874295017 +0000 UTC m=+866.605333175" watchObservedRunningTime="2026-07-02 13:28:11.876160726 +0000 UTC m=+866.607198867"
Jul 02 13:28:12.739602 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:12.739560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:28:13.740588 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:13.740547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:28:13.742339 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:13.742308    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:28:13.761489 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:13.761456    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd565435-bf6e-43df-b936-f3619e47476a" path="/var/lib/kubelet/pods/cd565435-bf6e-43df-b936-f3619e47476a/volumes"
Jul 02 13:28:23.739961 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:23.739913    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:28:33.740375 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:33.740331    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:28:43.740462 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:43.740366    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:28:45.783496 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:45.783456    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:28:45.784289 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:45.784257    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:28:53.739751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:28:53.739701    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:29:03.739890 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:29:03.739837    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:29:13.740255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:29:13.740204    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:29:23.740051 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:29:23.739986    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:29:33.740597 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:29:33.740547    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:29:43.740453 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:29:43.740396    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:29:53.739725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:29:53.739684    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:30:03.740762 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:03.740712    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:30:13.750673 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:13.750585    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:30:13.760669 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:13.760645    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:30:18.449906 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.449705    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"]
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.457184    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"]
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:30:18.461848    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:  > containerID="b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3" cmd=["/bin/sleep","15"]
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.461938    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" message="" err=<
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:  >
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:30:18.461957    2564 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="main" containerID="cri-o://b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"
Jul 02 13:30:18.463340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.462005    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="main" containerID="cri-o://b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3" gracePeriod=60
Jul 02 13:30:18.536548 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:30:18.536508    2564 logging.go:55] [core] [Channel #110 SubChannel #111]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.43:9003", ServerName: "10.132.0.43:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.43:9003: connect: connection refused"
Jul 02 13:30:18.722903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.722878    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:30:18.868147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-tokenizer-uds\") pod \"3e96e63f-0643-437c-8321-294005b83acd\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") "
Jul 02 13:30:18.868354 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hvtm\" (UniqueName: \"kubernetes.io/projected/3e96e63f-0643-437c-8321-294005b83acd-kube-api-access-7hvtm\") pod \"3e96e63f-0643-437c-8321-294005b83acd\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") "
Jul 02 13:30:18.868354 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868227    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e96e63f-0643-437c-8321-294005b83acd-tls-certs\") pod \"3e96e63f-0643-437c-8321-294005b83acd\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") "
Jul 02 13:30:18.868354 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-kserve-provision-location\") pod \"3e96e63f-0643-437c-8321-294005b83acd\" (UID: \"3e96e63f-0643-437c-8321-294005b83acd\") "
Jul 02 13:30:18.868553 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868371    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3e96e63f-0643-437c-8321-294005b83acd" (UID: "3e96e63f-0643-437c-8321-294005b83acd"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:18.868553 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868497    2564 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-tokenizer-uds\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:18.869006 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.868984    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e96e63f-0643-437c-8321-294005b83acd" (UID: "3e96e63f-0643-437c-8321-294005b83acd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:18.870396 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.870372    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e96e63f-0643-437c-8321-294005b83acd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e96e63f-0643-437c-8321-294005b83acd" (UID: "3e96e63f-0643-437c-8321-294005b83acd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:30:18.870509 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.870492    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e96e63f-0643-437c-8321-294005b83acd-kube-api-access-7hvtm" (OuterVolumeSpecName: "kube-api-access-7hvtm") pod "3e96e63f-0643-437c-8321-294005b83acd" (UID: "3e96e63f-0643-437c-8321-294005b83acd"). InnerVolumeSpecName "kube-api-access-7hvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:30:18.969584 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.969504    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hvtm\" (UniqueName: \"kubernetes.io/projected/3e96e63f-0643-437c-8321-294005b83acd-kube-api-access-7hvtm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:18.969584 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.969570    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e96e63f-0643-437c-8321-294005b83acd-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:18.969584 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:18.969582    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e96e63f-0643-437c-8321-294005b83acd-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:19.532151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.532110    2564 generic.go:356] "Generic (PLEG): container finished" podID="3e96e63f-0643-437c-8321-294005b83acd" containerID="b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3" exitCode=0
Jul 02 13:30:19.532663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.532167    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"
Jul 02 13:30:19.532663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.532198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" event={"ID":"3e96e63f-0643-437c-8321-294005b83acd","Type":"ContainerDied","Data":"b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"}
Jul 02 13:30:19.532663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.532254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" event={"ID":"3e96e63f-0643-437c-8321-294005b83acd","Type":"ContainerDied","Data":"35f91947f1195969f17a7a04dd8e9050201e0831954127e1a69ed5f5f27fcfe9"}
Jul 02 13:30:19.532663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.532278    2564 scope.go:117] "RemoveContainer" containerID="b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"
Jul 02 13:30:19.536244 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.536211    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.43:9003\" within 1s: context deadline exceeded"
Jul 02 13:30:19.541759 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.541739    2564 scope.go:117] "RemoveContainer" containerID="8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6"
Jul 02 13:30:19.549674 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.549655    2564 scope.go:117] "RemoveContainer" containerID="b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"
Jul 02 13:30:19.549916 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:30:19.549898    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3\": container with ID starting with b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3 not found: ID does not exist" containerID="b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"
Jul 02 13:30:19.549984 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.549922    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3"} err="failed to get container status \"b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3\": rpc error: code = NotFound desc = could not find container \"b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3\": container with ID starting with b56861a2e710c3e36fde097743b368f24e3f4d2502d347e9d83707ce229b4bc3 not found: ID does not exist"
Jul 02 13:30:19.549984 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.549941    2564 scope.go:117] "RemoveContainer" containerID="8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6"
Jul 02 13:30:19.550167 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:30:19.550152    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6\": container with ID starting with 8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6 not found: ID does not exist" containerID="8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6"
Jul 02 13:30:19.550208 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.550170    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6"} err="failed to get container status \"8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6\": rpc error: code = NotFound desc = could not find container \"8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6\": container with ID starting with 8081836ec0d6e6a96c54af7a28af1176e3f3430d07ba052a63ed7783ae4685d6 not found: ID does not exist"
Jul 02 13:30:19.556779 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.556745    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"]
Jul 02 13:30:19.563013 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.562992    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg"]
Jul 02 13:30:19.761633 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:19.761603    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e96e63f-0643-437c-8321-294005b83acd" path="/var/lib/kubelet/pods/3e96e63f-0643-437c-8321-294005b83acd/volumes"
Jul 02 13:30:33.485416 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:33.485348    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" containerID="cri-o://d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395" gracePeriod=45
Jul 02 13:30:33.751685 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:33.751571    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 02 13:30:34.241034 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.241009    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:30:34.403289 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403258    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-model-cache\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403295    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-kserve-provision-location\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403323    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4hz6\" (UniqueName: \"kubernetes.io/projected/10c8a29f-0e04-498c-ab4a-e2a910726d24-kube-api-access-j4hz6\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403352    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-tmp-dir\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-home\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403409    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-dshm\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c8a29f-0e04-498c-ab4a-e2a910726d24-tls-certs\") pod \"10c8a29f-0e04-498c-ab4a-e2a910726d24\" (UID: \"10c8a29f-0e04-498c-ab4a-e2a910726d24\") "
Jul 02 13:30:34.403818 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403600    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:34.403818 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.403797    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.404119 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.404065    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-home" (OuterVolumeSpecName: "home") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:34.406158 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.406122    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-dshm" (OuterVolumeSpecName: "dshm") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:34.406158 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.406129    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c8a29f-0e04-498c-ab4a-e2a910726d24-kube-api-access-j4hz6" (OuterVolumeSpecName: "kube-api-access-j4hz6") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "kube-api-access-j4hz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:30:34.406293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.406151    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c8a29f-0e04-498c-ab4a-e2a910726d24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:30:34.416438 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.416414    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:34.459410 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.459377    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10c8a29f-0e04-498c-ab4a-e2a910726d24" (UID: "10c8a29f-0e04-498c-ab4a-e2a910726d24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:30:34.504290 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.504261    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.504290 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.504287    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.504688 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.504298    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c8a29f-0e04-498c-ab4a-e2a910726d24-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.504688 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.504314    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.504688 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.504325    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hz6\" (UniqueName: \"kubernetes.io/projected/10c8a29f-0e04-498c-ab4a-e2a910726d24-kube-api-access-j4hz6\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.504688 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.504339    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c8a29f-0e04-498c-ab4a-e2a910726d24-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:30:34.610058 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.610028    2564 generic.go:356] "Generic (PLEG): container finished" podID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerID="d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395" exitCode=0
Jul 02 13:30:34.610265 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.610092    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"
Jul 02 13:30:34.610265 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.610123    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" event={"ID":"10c8a29f-0e04-498c-ab4a-e2a910726d24","Type":"ContainerDied","Data":"d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395"}
Jul 02 13:30:34.610265 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.610187    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g" event={"ID":"10c8a29f-0e04-498c-ab4a-e2a910726d24","Type":"ContainerDied","Data":"3e3f93a5a1c906d6b2f8309ed0fca2a1ac83b1ddfaf5b29ecb367301e4748d16"}
Jul 02 13:30:34.610265 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.610209    2564 scope.go:117] "RemoveContainer" containerID="d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395"
Jul 02 13:30:34.619280 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.619263    2564 scope.go:117] "RemoveContainer" containerID="09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59"
Jul 02 13:30:34.627489 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.627473    2564 scope.go:117] "RemoveContainer" containerID="d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395"
Jul 02 13:30:34.627779 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:30:34.627762    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395\": container with ID starting with d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395 not found: ID does not exist" containerID="d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395"
Jul 02 13:30:34.627839 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.627787    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395"} err="failed to get container status \"d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395\": rpc error: code = NotFound desc = could not find container \"d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395\": container with ID starting with d8585b6c6a0aba6ad4de1898413d3aa2099ac1bdb17154b6a6b156063da66395 not found: ID does not exist"
Jul 02 13:30:34.627839 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.627805    2564 scope.go:117] "RemoveContainer" containerID="09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59"
Jul 02 13:30:34.628038 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:30:34.628021    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59\": container with ID starting with 09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59 not found: ID does not exist" containerID="09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59"
Jul 02 13:30:34.628075 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.628044    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59"} err="failed to get container status \"09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59\": rpc error: code = NotFound desc = could not find container \"09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59\": container with ID starting with 09534bfc6eb23d5b2a500e1f16db7f1995b71ede22effb1cd7b4765183d76d59 not found: ID does not exist"
Jul 02 13:30:34.637186 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.637166    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"]
Jul 02 13:30:34.642419 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:34.642398    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g"]
Jul 02 13:30:35.760222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:35.760189    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" path="/var/lib/kubelet/pods/10c8a29f-0e04-498c-ab4a-e2a910726d24/volumes"
Jul 02 13:30:44.284152 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284117    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"]
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284636    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="storage-initializer"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284661    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="storage-initializer"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284672    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="main"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284681    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="main"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284702    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd565435-bf6e-43df-b936-f3619e47476a" containerName="main"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284711    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd565435-bf6e-43df-b936-f3619e47476a" containerName="main"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284732    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="storage-initializer"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284741    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="storage-initializer"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284753    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd565435-bf6e-43df-b936-f3619e47476a" containerName="storage-initializer"
Jul 02 13:30:44.284766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284760    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd565435-bf6e-43df-b936-f3619e47476a" containerName="storage-initializer"
Jul 02 13:30:44.285324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284777    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main"
Jul 02 13:30:44.285324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284785    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main"
Jul 02 13:30:44.285324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284871    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd565435-bf6e-43df-b936-f3619e47476a" containerName="main"
Jul 02 13:30:44.285324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284883    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e96e63f-0643-437c-8321-294005b83acd" containerName="main"
Jul 02 13:30:44.285324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.284893    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c8a29f-0e04-498c-ab4a-e2a910726d24" containerName="main"
Jul 02 13:30:44.288151 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.288127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.292589 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.292561    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:30:44.292747 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.292609    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:30:44.292747 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.292636    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:30:44.292747 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.292608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 02 13:30:44.292910 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.292892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:30:44.299514 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.299486    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"]
Jul 02 13:30:44.385054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385017    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-tmp-dir\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.385054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-dshm\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.385299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.385299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-model-cache\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.385299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzc5b\" (UniqueName: \"kubernetes.io/projected/843adbdd-ca16-4f1a-8178-590ef36114b3-kube-api-access-qzc5b\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.385299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-home\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.385299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.385203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843adbdd-ca16-4f1a-8178-590ef36114b3-tls-certs\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486400 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843adbdd-ca16-4f1a-8178-590ef36114b3-tls-certs\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-tmp-dir\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-dshm\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-model-cache\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzc5b\" (UniqueName: \"kubernetes.io/projected/843adbdd-ca16-4f1a-8178-590ef36114b3-kube-api-access-qzc5b\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486920 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-home\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.486920 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.486836    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-tmp-dir\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.487023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.487003    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-home\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.487063 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.487043    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-model-cache\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.487110 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.487079    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.488876 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.488852    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-dshm\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.489146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.489128    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843adbdd-ca16-4f1a-8178-590ef36114b3-tls-certs\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.495776 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.495752    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzc5b\" (UniqueName: \"kubernetes.io/projected/843adbdd-ca16-4f1a-8178-590ef36114b3-kube-api-access-qzc5b\") pod \"auth-invalid-token-test-kserve-54f4fd4b56-mk7fx\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.599170 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.599039    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:44.624359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.624317    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"]
Jul 02 13:30:44.628492 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.628467    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.632226 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.632023    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-7kcnq\""
Jul 02 13:30:44.640245 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.640217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"]
Jul 02 13:30:44.689256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.689224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.689413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.689281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.689413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.689312    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrvf\" (UniqueName: \"kubernetes.io/projected/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kube-api-access-twrvf\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.689413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.689378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.754674 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.754635    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"]
Jul 02 13:30:44.756036 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:30:44.756008    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843adbdd_ca16_4f1a_8178_590ef36114b3.slice/crio-37f0cf59eabc966272b31d3443bfabd5eb985a3fe26471c564125377b16f9f08 WatchSource:0}: Error finding container 37f0cf59eabc966272b31d3443bfabd5eb985a3fe26471c564125377b16f9f08: Status 404 returned error can't find the container with id 37f0cf59eabc966272b31d3443bfabd5eb985a3fe26471c564125377b16f9f08
Jul 02 13:30:44.790494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.790464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.790622 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.790508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrvf\" (UniqueName: \"kubernetes.io/projected/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kube-api-access-twrvf\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.790709 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.790691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.790842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.790819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.791144 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.791124    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.791203 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.791117    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.792988 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.792968    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.799445 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.799424    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrvf\" (UniqueName: \"kubernetes.io/projected/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kube-api-access-twrvf\") pod \"auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:44.939982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:44.939941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:45.096698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:45.096674    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"]
Jul 02 13:30:45.098329 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:30:45.098303    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ced59c7_3dc7_4cab_a386_d1c448c7a65e.slice/crio-b5eefb2beab0613f4d376d74e705e0a796b29302e02e4fe8f7c9a6b02536cfff WatchSource:0}: Error finding container b5eefb2beab0613f4d376d74e705e0a796b29302e02e4fe8f7c9a6b02536cfff: Status 404 returned error can't find the container with id b5eefb2beab0613f4d376d74e705e0a796b29302e02e4fe8f7c9a6b02536cfff
Jul 02 13:30:45.674265 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:45.674188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" event={"ID":"1ced59c7-3dc7-4cab-a386-d1c448c7a65e","Type":"ContainerStarted","Data":"cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d"}
Jul 02 13:30:45.674265 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:45.674275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" event={"ID":"1ced59c7-3dc7-4cab-a386-d1c448c7a65e","Type":"ContainerStarted","Data":"b5eefb2beab0613f4d376d74e705e0a796b29302e02e4fe8f7c9a6b02536cfff"}
Jul 02 13:30:45.676387 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:45.676355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" event={"ID":"843adbdd-ca16-4f1a-8178-590ef36114b3","Type":"ContainerStarted","Data":"0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be"}
Jul 02 13:30:45.676561 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:45.676413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" event={"ID":"843adbdd-ca16-4f1a-8178-590ef36114b3","Type":"ContainerStarted","Data":"37f0cf59eabc966272b31d3443bfabd5eb985a3fe26471c564125377b16f9f08"}
Jul 02 13:30:46.690731 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:46.690689    2564 generic.go:356] "Generic (PLEG): container finished" podID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerID="cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d" exitCode=0
Jul 02 13:30:46.691227 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:46.690779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" event={"ID":"1ced59c7-3dc7-4cab-a386-d1c448c7a65e","Type":"ContainerDied","Data":"cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d"}
Jul 02 13:30:47.698969 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:47.698930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" event={"ID":"1ced59c7-3dc7-4cab-a386-d1c448c7a65e","Type":"ContainerStarted","Data":"9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"}
Jul 02 13:30:47.699422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:47.699008    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:30:47.720288 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:47.720228    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" podStartSLOduration=3.024187071 podStartE2EDuration="3.720211555s" totalImagesPullingTime="0s" totalInitContainerRuntime="696.024484ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:47.718492606 +0000 UTC m=+1022.449530781" watchObservedRunningTime="2026-07-02 13:30:47.720211555 +0000 UTC m=+1022.451249729"
Jul 02 13:30:49.710815 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:49.710778    2564 generic.go:356] "Generic (PLEG): container finished" podID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerID="0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be" exitCode=0
Jul 02 13:30:49.711206 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:49.710854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" event={"ID":"843adbdd-ca16-4f1a-8178-590ef36114b3","Type":"ContainerDied","Data":"0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be"}
Jul 02 13:30:50.717745 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:50.717706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" event={"ID":"843adbdd-ca16-4f1a-8178-590ef36114b3","Type":"ContainerStarted","Data":"412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb"}
Jul 02 13:30:50.743918 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:50.743860    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podStartSLOduration=2.641434328 podStartE2EDuration="6.743842359s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.102408031s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:50.740976938 +0000 UTC m=+1025.472015090" watchObservedRunningTime="2026-07-02 13:30:50.743842359 +0000 UTC m=+1025.474880501"
Jul 02 13:30:51.599813 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:51.599772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:54.599956 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:54.599919    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:30:54.601650 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:30:54.601617    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:31:04.600079 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:04.600026    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:31:14.599714 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:14.599666    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:31:18.705991 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:18.705961    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:31:24.599633 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:24.599595    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:31:34.599955 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:34.599905    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:31:44.599768 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:44.599666    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:31:54.600352 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:31:54.600304    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:32:04.600438 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:04.600379    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:32:14.599512 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:14.599457    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:32:24.599708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:24.599663    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:32:34.599773 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:34.599724    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 02 13:32:44.609547 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:44.609490    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:32:44.617189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:44.617163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:32:54.904830 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:54.904793    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"]
Jul 02 13:32:54.911835 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:54.911805    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"]
Jul 02 13:32:54.920920 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:32:54.920881    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:32:54.920920 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:32:54.920920 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:32:54.920920 ip-10-0-138-135 kubenswrapper[2564]:  > containerID="9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9" cmd=["/bin/sleep","15"]
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:54.920935    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" message="" err=<
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]:  >
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:32:54.920948    2564 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="main" containerID="cri-o://9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"
Jul 02 13:32:54.921262 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:54.920986    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="main" containerID="cri-o://9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9" gracePeriod=60
Jul 02 13:32:55.193743 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.193717    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:32:55.374785 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.374750    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twrvf\" (UniqueName: \"kubernetes.io/projected/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kube-api-access-twrvf\") pod \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") "
Jul 02 13:32:55.375037 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.374852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kserve-provision-location\") pod \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") "
Jul 02 13:32:55.375037 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.374901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tls-certs\") pod \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") "
Jul 02 13:32:55.375037 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.374916    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tokenizer-uds\") pod \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\" (UID: \"1ced59c7-3dc7-4cab-a386-d1c448c7a65e\") "
Jul 02 13:32:55.375270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.375233    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1ced59c7-3dc7-4cab-a386-d1c448c7a65e" (UID: "1ced59c7-3dc7-4cab-a386-d1c448c7a65e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:55.375646 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.375624    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ced59c7-3dc7-4cab-a386-d1c448c7a65e" (UID: "1ced59c7-3dc7-4cab-a386-d1c448c7a65e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:55.377060 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.377034    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ced59c7-3dc7-4cab-a386-d1c448c7a65e" (UID: "1ced59c7-3dc7-4cab-a386-d1c448c7a65e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:32:55.377173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.377127    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kube-api-access-twrvf" (OuterVolumeSpecName: "kube-api-access-twrvf") pod "1ced59c7-3dc7-4cab-a386-d1c448c7a65e" (UID: "1ced59c7-3dc7-4cab-a386-d1c448c7a65e"). InnerVolumeSpecName "kube-api-access-twrvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:32:55.398329 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.398298    2564 generic.go:356] "Generic (PLEG): container finished" podID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerID="9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9" exitCode=0
Jul 02 13:32:55.398440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.398359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" event={"ID":"1ced59c7-3dc7-4cab-a386-d1c448c7a65e","Type":"ContainerDied","Data":"9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"}
Jul 02 13:32:55.398440 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.398423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr" event={"ID":"1ced59c7-3dc7-4cab-a386-d1c448c7a65e","Type":"ContainerDied","Data":"b5eefb2beab0613f4d376d74e705e0a796b29302e02e4fe8f7c9a6b02536cfff"}
Jul 02 13:32:55.398563 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.398446    2564 scope.go:117] "RemoveContainer" containerID="9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"
Jul 02 13:32:55.398563 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.398371    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"
Jul 02 13:32:55.407769 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.407752    2564 scope.go:117] "RemoveContainer" containerID="cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d"
Jul 02 13:32:55.415654 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.415589    2564 scope.go:117] "RemoveContainer" containerID="9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"
Jul 02 13:32:55.415889 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:32:55.415868    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9\": container with ID starting with 9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9 not found: ID does not exist" containerID="9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"
Jul 02 13:32:55.415943 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.415897    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9"} err="failed to get container status \"9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9\": rpc error: code = NotFound desc = could not find container \"9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9\": container with ID starting with 9bf0215bf679f95a2792f5c01a7b63ac641a7fa6b4ad0c87017d364813d030e9 not found: ID does not exist"
Jul 02 13:32:55.415943 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.415916    2564 scope.go:117] "RemoveContainer" containerID="cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d"
Jul 02 13:32:55.416165 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:32:55.416147    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d\": container with ID starting with cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d not found: ID does not exist" containerID="cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d"
Jul 02 13:32:55.416207 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.416171    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d"} err="failed to get container status \"cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d\": rpc error: code = NotFound desc = could not find container \"cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d\": container with ID starting with cce843fdcedbbe1b1c42e1b27b5ccce73701f820d4bd7f64038a03dd9bfc815d not found: ID does not exist"
Jul 02 13:32:55.421822 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.421795    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"]
Jul 02 13:32:55.423878 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.423823    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr"]
Jul 02 13:32:55.476079 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.476049    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:32:55.476079 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.476081    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:32:55.476270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.476095    2564 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-tokenizer-uds\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:32:55.476270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.476107    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twrvf\" (UniqueName: \"kubernetes.io/projected/1ced59c7-3dc7-4cab-a386-d1c448c7a65e-kube-api-access-twrvf\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:32:55.760898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:32:55.760815    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" path="/var/lib/kubelet/pods/1ced59c7-3dc7-4cab-a386-d1c448c7a65e/volumes"
Jul 02 13:33:09.931923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:09.931821    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" containerID="cri-o://412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb" gracePeriod=45
Jul 02 13:33:10.691952 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.691924    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:33:10.696902 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.696880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-model-cache\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697034 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.696919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-dshm\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697034 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.696970    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-tmp-dir\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697034 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.696997    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-home\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697034 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.697014    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843adbdd-ca16-4f1a-8178-590ef36114b3-tls-certs\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697259 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.697043    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzc5b\" (UniqueName: \"kubernetes.io/projected/843adbdd-ca16-4f1a-8178-590ef36114b3-kube-api-access-qzc5b\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697259 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.697085    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-kserve-provision-location\") pod \"843adbdd-ca16-4f1a-8178-590ef36114b3\" (UID: \"843adbdd-ca16-4f1a-8178-590ef36114b3\") "
Jul 02 13:33:10.697259 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.697169    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:10.697415 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.697311    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.697803 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.697777    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-home" (OuterVolumeSpecName: "home") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:10.699555 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.699507    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843adbdd-ca16-4f1a-8178-590ef36114b3-kube-api-access-qzc5b" (OuterVolumeSpecName: "kube-api-access-qzc5b") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "kube-api-access-qzc5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:33:10.699716 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.699679    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843adbdd-ca16-4f1a-8178-590ef36114b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:33:10.699831 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.699774    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-dshm" (OuterVolumeSpecName: "dshm") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:10.715865 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.715800    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:10.760633 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.760591    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "843adbdd-ca16-4f1a-8178-590ef36114b3" (UID: "843adbdd-ca16-4f1a-8178-590ef36114b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:10.798183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.798150    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.798183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.798178    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.798183 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.798188    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843adbdd-ca16-4f1a-8178-590ef36114b3-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.798405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.798197    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzc5b\" (UniqueName: \"kubernetes.io/projected/843adbdd-ca16-4f1a-8178-590ef36114b3-kube-api-access-qzc5b\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.798405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.798207    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.798405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:10.798215    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/843adbdd-ca16-4f1a-8178-590ef36114b3-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:33:11.484567 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.484509    2564 generic.go:356] "Generic (PLEG): container finished" podID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerID="412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb" exitCode=0
Jul 02 13:33:11.484982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.484596    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"
Jul 02 13:33:11.484982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.484600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" event={"ID":"843adbdd-ca16-4f1a-8178-590ef36114b3","Type":"ContainerDied","Data":"412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb"}
Jul 02 13:33:11.484982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.484662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" event={"ID":"843adbdd-ca16-4f1a-8178-590ef36114b3","Type":"ContainerDied","Data":"37f0cf59eabc966272b31d3443bfabd5eb985a3fe26471c564125377b16f9f08"}
Jul 02 13:33:11.484982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.484687    2564 scope.go:117] "RemoveContainer" containerID="412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb"
Jul 02 13:33:11.494198 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.494179    2564 scope.go:117] "RemoveContainer" containerID="0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be"
Jul 02 13:33:11.503025 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.503005    2564 scope.go:117] "RemoveContainer" containerID="412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb"
Jul 02 13:33:11.503290 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:33:11.503267    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb\": container with ID starting with 412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb not found: ID does not exist" containerID="412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb"
Jul 02 13:33:11.503374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.503307    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb"} err="failed to get container status \"412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb\": rpc error: code = NotFound desc = could not find container \"412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb\": container with ID starting with 412129bebf486b8a7601918993e3b0b4326e3804e9031b26fbd7e055afe100cb not found: ID does not exist"
Jul 02 13:33:11.503374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.503333    2564 scope.go:117] "RemoveContainer" containerID="0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be"
Jul 02 13:33:11.503619 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:33:11.503600    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be\": container with ID starting with 0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be not found: ID does not exist" containerID="0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be"
Jul 02 13:33:11.503662 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.503626    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be"} err="failed to get container status \"0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be\": rpc error: code = NotFound desc = could not find container \"0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be\": container with ID starting with 0b5d29654c4f9806c1d94896c4b159345e139ab17ed3e0c7f8492529440420be not found: ID does not exist"
Jul 02 13:33:11.514004 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.509946    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"]
Jul 02 13:33:11.516923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.516890    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx"]
Jul 02 13:33:11.611387 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.611335    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 02 13:33:11.759995 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:11.759899    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" path="/var/lib/kubelet/pods/843adbdd-ca16-4f1a-8178-590ef36114b3/volumes"
Jul 02 13:33:45.814660 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:45.814633    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:33:45.817165 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:45.817142    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:33:47.332827 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.332784    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"]
Jul 02 13:33:47.333214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333164    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="storage-initializer"
Jul 02 13:33:47.333214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333177    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="storage-initializer"
Jul 02 13:33:47.333214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333190    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="main"
Jul 02 13:33:47.333214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333196    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="main"
Jul 02 13:33:47.333214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333209    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main"
Jul 02 13:33:47.333214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333215    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main"
Jul 02 13:33:47.333477 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333222    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="storage-initializer"
Jul 02 13:33:47.333477 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333227    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="storage-initializer"
Jul 02 13:33:47.333477 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333279    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="843adbdd-ca16-4f1a-8178-590ef36114b3" containerName="main"
Jul 02 13:33:47.333477 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.333290    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ced59c7-3dc7-4cab-a386-d1c448c7a65e" containerName="main"
Jul 02 13:33:47.337014 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.336997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.340252 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.340229    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:33:47.340380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.340229    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:33:47.340380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.340299    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 02 13:33:47.340573 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.340560    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:33:47.341397 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.341381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:33:47.350304 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.350279    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"]
Jul 02 13:33:47.399106 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcprp\" (UniqueName: \"kubernetes.io/projected/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kube-api-access-qcprp\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.399246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-dshm\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.399246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kserve-provision-location\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.399246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-home\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.399246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tls-certs\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.399246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tmp-dir\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.399418 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.399286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-model-cache\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500410 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcprp\" (UniqueName: \"kubernetes.io/projected/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kube-api-access-qcprp\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500410 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-dshm\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500440    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kserve-provision-location\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-home\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tls-certs\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tmp-dir\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-model-cache\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500883 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500856    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kserve-provision-location\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.500949 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500903    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-home\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.501008 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.500973    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tmp-dir\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.501090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.501070    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-model-cache\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.502774 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.502749    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-dshm\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.503068 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.503047    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tls-certs\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.509964 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.509945    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcprp\" (UniqueName: \"kubernetes.io/projected/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kube-api-access-qcprp\") pod \"auth-disabled-test-kserve-79f6d89dd6-pslkr\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.567979 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.567945    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"]
Jul 02 13:33:47.571718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.571696    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.574509 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.574491    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-b696c\""
Jul 02 13:33:47.583847 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.583820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"]
Jul 02 13:33:47.647690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.647665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:47.703729 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.703381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.703729 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.703454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.703729 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.703519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.703729 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.703609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjz4\" (UniqueName: \"kubernetes.io/projected/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kube-api-access-jwjz4\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.792726 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.792700    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"]
Jul 02 13:33:47.794063 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:33:47.794031    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8006303f_68d9_4415_86f4_6cdc3c27e3b8.slice/crio-e105b40e5c66fbba14d658d7361192da3b4a6b78975b85380f14fb2c20f103a1 WatchSource:0}: Error finding container e105b40e5c66fbba14d658d7361192da3b4a6b78975b85380f14fb2c20f103a1: Status 404 returned error can't find the container with id e105b40e5c66fbba14d658d7361192da3b4a6b78975b85380f14fb2c20f103a1
Jul 02 13:33:47.806853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.804859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.806853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.804925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.806853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.804982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjz4\" (UniqueName: \"kubernetes.io/projected/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kube-api-access-jwjz4\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.806853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.805035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.807289 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.807260    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.807396 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.807273    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.808132 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.808110    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.815201 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.815176    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjz4\" (UniqueName: \"kubernetes.io/projected/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kube-api-access-jwjz4\") pod \"auth-disabled-test-kserve-router-scheduler-795c57555-rhs74\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:47.881699 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:47.881664    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:48.023733 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:48.023656    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"]
Jul 02 13:33:48.026326 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:33:48.026299    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5389ab4_790d_47d7_8d73_95d0d8edf24e.slice/crio-8c10ca042824c18e6d8f043b89e531bac540ec639f267c49ebfc18dc66e41003 WatchSource:0}: Error finding container 8c10ca042824c18e6d8f043b89e531bac540ec639f267c49ebfc18dc66e41003: Status 404 returned error can't find the container with id 8c10ca042824c18e6d8f043b89e531bac540ec639f267c49ebfc18dc66e41003
Jul 02 13:33:48.675733 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:48.675688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" event={"ID":"8006303f-68d9-4415-86f4-6cdc3c27e3b8","Type":"ContainerStarted","Data":"7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827"}
Jul 02 13:33:48.676205 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:48.675757    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" event={"ID":"8006303f-68d9-4415-86f4-6cdc3c27e3b8","Type":"ContainerStarted","Data":"e105b40e5c66fbba14d658d7361192da3b4a6b78975b85380f14fb2c20f103a1"}
Jul 02 13:33:48.677734 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:48.677707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" event={"ID":"e5389ab4-790d-47d7-8d73-95d0d8edf24e","Type":"ContainerStarted","Data":"10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a"}
Jul 02 13:33:48.678083 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:48.677756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" event={"ID":"e5389ab4-790d-47d7-8d73-95d0d8edf24e","Type":"ContainerStarted","Data":"8c10ca042824c18e6d8f043b89e531bac540ec639f267c49ebfc18dc66e41003"}
Jul 02 13:33:48.810101 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:33:48.810069    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5389ab4_790d_47d7_8d73_95d0d8edf24e.slice/crio-conmon-10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a.scope\": RecentStats: unable to find data in memory cache]"
Jul 02 13:33:49.687404 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:49.687356    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerID="10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a" exitCode=0
Jul 02 13:33:49.687970 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:49.687497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" event={"ID":"e5389ab4-790d-47d7-8d73-95d0d8edf24e","Type":"ContainerDied","Data":"10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a"}
Jul 02 13:33:50.695677 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:50.695636    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" event={"ID":"e5389ab4-790d-47d7-8d73-95d0d8edf24e","Type":"ContainerStarted","Data":"5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"}
Jul 02 13:33:50.696067 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:50.695772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:33:50.715379 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:50.715321    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" podStartSLOduration=3.035254399 podStartE2EDuration="3.715303061s" totalImagesPullingTime="0s" totalInitContainerRuntime="680.048662ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:33:50.714767651 +0000 UTC m=+1205.445805827" watchObservedRunningTime="2026-07-02 13:33:50.715303061 +0000 UTC m=+1205.446341203"
Jul 02 13:33:52.708203 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:52.708164    2564 generic.go:356] "Generic (PLEG): container finished" podID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerID="7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827" exitCode=0
Jul 02 13:33:52.708686 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:52.708249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" event={"ID":"8006303f-68d9-4415-86f4-6cdc3c27e3b8","Type":"ContainerDied","Data":"7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827"}
Jul 02 13:33:53.716494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:53.716457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" event={"ID":"8006303f-68d9-4415-86f4-6cdc3c27e3b8","Type":"ContainerStarted","Data":"9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d"}
Jul 02 13:33:53.741610 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:53.741549    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podStartSLOduration=2.776046565 podStartE2EDuration="6.741514871s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.965468306s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:33:53.739571133 +0000 UTC m=+1208.470609284" watchObservedRunningTime="2026-07-02 13:33:53.741514871 +0000 UTC m=+1208.472553001"
Jul 02 13:33:54.648473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:54.648440    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:57.648602 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:57.648555    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:33:57.650585 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:33:57.650548    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:34:07.648746 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:07.648696    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:34:17.648611 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:17.648556    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:34:21.703256 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:21.703220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:34:27.648110 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:27.648062    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:34:37.648231 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:37.648184    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:34:47.649021 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:47.648962    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:34:57.648961 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:34:57.648903    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:35:07.649131 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:07.649071    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:35:17.648332 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:17.648282    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:35:27.648619 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:27.648569    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:35:36.120885 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.120850    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"]
Jul 02 13:35:36.131377 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.131296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.134905 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.134869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:35:36.135072 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.134928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 02 13:35:36.146958 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.146931    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"]
Jul 02 13:35:36.279080 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279080 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m589\" (UniqueName: \"kubernetes.io/projected/53e9be58-570d-4207-a000-4ce1a62d2537-kube-api-access-7m589\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9be58-570d-4207-a000-4ce1a62d2537-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279296    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279441 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.279441 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.279346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9be58-570d-4207-a000-4ce1a62d2537-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380534 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9be58-570d-4207-a000-4ce1a62d2537-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380703 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380703 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380703 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9be58-570d-4207-a000-4ce1a62d2537-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380916 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380916 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380916 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.380916 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.380880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m589\" (UniqueName: \"kubernetes.io/projected/53e9be58-570d-4207-a000-4ce1a62d2537-kube-api-access-7m589\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.381145 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.381029    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.381296 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.381245    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.381446 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.381416    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.381598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.381324    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9be58-570d-4207-a000-4ce1a62d2537-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.381598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.381316    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.383292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.383273    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9be58-570d-4207-a000-4ce1a62d2537-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.383428 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.383409    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.390678 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.390655    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m589\" (UniqueName: \"kubernetes.io/projected/53e9be58-570d-4207-a000-4ce1a62d2537-kube-api-access-7m589\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.443550 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.443484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:36.612204 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:36.612176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"]
Jul 02 13:35:36.614969 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:35:36.614931    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e9be58_570d_4207_a000_4ce1a62d2537.slice/crio-c62f8b9ffe1d378e0820f2267f7ccb44c2345ee9e1031a9dae63b19ff8089cb9 WatchSource:0}: Error finding container c62f8b9ffe1d378e0820f2267f7ccb44c2345ee9e1031a9dae63b19ff8089cb9: Status 404 returned error can't find the container with id c62f8b9ffe1d378e0820f2267f7ccb44c2345ee9e1031a9dae63b19ff8089cb9
Jul 02 13:35:37.298664 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:37.298616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" event={"ID":"53e9be58-570d-4207-a000-4ce1a62d2537","Type":"ContainerStarted","Data":"bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e"}
Jul 02 13:35:37.299157 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:37.298688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" event={"ID":"53e9be58-570d-4207-a000-4ce1a62d2537","Type":"ContainerStarted","Data":"c62f8b9ffe1d378e0820f2267f7ccb44c2345ee9e1031a9dae63b19ff8089cb9"}
Jul 02 13:35:37.649189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:37.649136    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:35:41.323017 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:41.322978    2564 generic.go:356] "Generic (PLEG): container finished" podID="53e9be58-570d-4207-a000-4ce1a62d2537" containerID="bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e" exitCode=0
Jul 02 13:35:41.323647 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:41.323065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" event={"ID":"53e9be58-570d-4207-a000-4ce1a62d2537","Type":"ContainerDied","Data":"bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e"}
Jul 02 13:35:42.330237 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:42.330190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" event={"ID":"53e9be58-570d-4207-a000-4ce1a62d2537","Type":"ContainerStarted","Data":"09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154"}
Jul 02 13:35:42.360414 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:42.360355    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podStartSLOduration=2.182176166 podStartE2EDuration="6.360338361s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.178162195s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:35:42.357219783 +0000 UTC m=+1317.088257938" watchObservedRunningTime="2026-07-02 13:35:42.360338361 +0000 UTC m=+1317.091376490"
Jul 02 13:35:42.444618 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:42.444585    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:46.444619 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:46.444580    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:35:46.446475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:46.446443    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:35:47.658989 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:47.658956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:35:47.667098 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:47.667065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:35:56.444580 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:35:56.444518    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:36:04.005041 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.004941    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"]
Jul 02 13:36:04.007753 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.007712    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"]
Jul 02 13:36:04.019105 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:36:04.019059    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:36:04.019105 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:36:04.019105 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:36:04.019105 ip-10-0-138-135 kubenswrapper[2564]:  > containerID="5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01" cmd=["/bin/sleep","15"]
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.019119    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" message="" err=<
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]:  >
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:36:04.019131    2564 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="main" containerID="cri-o://5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"
Jul 02 13:36:04.019475 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.019169    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="main" containerID="cri-o://5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01" gracePeriod=60
Jul 02 13:36:04.280782 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.280754    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:36:04.339283 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.339248    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kserve-provision-location\") pod \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") "
Jul 02 13:36:04.339457 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.339299    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tokenizer-uds\") pod \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") "
Jul 02 13:36:04.339457 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.339359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwjz4\" (UniqueName: \"kubernetes.io/projected/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kube-api-access-jwjz4\") pod \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") "
Jul 02 13:36:04.339457 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.339426    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tls-certs\") pod \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\" (UID: \"e5389ab4-790d-47d7-8d73-95d0d8edf24e\") "
Jul 02 13:36:04.339700 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.339671    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e5389ab4-790d-47d7-8d73-95d0d8edf24e" (UID: "e5389ab4-790d-47d7-8d73-95d0d8edf24e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:04.340169 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.340138    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5389ab4-790d-47d7-8d73-95d0d8edf24e" (UID: "e5389ab4-790d-47d7-8d73-95d0d8edf24e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:04.341749 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.341719    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5389ab4-790d-47d7-8d73-95d0d8edf24e" (UID: "e5389ab4-790d-47d7-8d73-95d0d8edf24e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:36:04.341843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.341759    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kube-api-access-jwjz4" (OuterVolumeSpecName: "kube-api-access-jwjz4") pod "e5389ab4-790d-47d7-8d73-95d0d8edf24e" (UID: "e5389ab4-790d-47d7-8d73-95d0d8edf24e"). InnerVolumeSpecName "kube-api-access-jwjz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:36:04.440374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.440335    2564 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tokenizer-uds\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:04.440374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.440363    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwjz4\" (UniqueName: \"kubernetes.io/projected/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kube-api-access-jwjz4\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:04.440374 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.440374    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5389ab4-790d-47d7-8d73-95d0d8edf24e-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:04.440626 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.440383    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5389ab4-790d-47d7-8d73-95d0d8edf24e-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:04.466676 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.466637    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerID="5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01" exitCode=0
Jul 02 13:36:04.466843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.466691    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"
Jul 02 13:36:04.466843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.466724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" event={"ID":"e5389ab4-790d-47d7-8d73-95d0d8edf24e","Type":"ContainerDied","Data":"5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"}
Jul 02 13:36:04.466843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.466779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74" event={"ID":"e5389ab4-790d-47d7-8d73-95d0d8edf24e","Type":"ContainerDied","Data":"8c10ca042824c18e6d8f043b89e531bac540ec639f267c49ebfc18dc66e41003"}
Jul 02 13:36:04.466843 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.466800    2564 scope.go:117] "RemoveContainer" containerID="5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"
Jul 02 13:36:04.477761 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.477742    2564 scope.go:117] "RemoveContainer" containerID="10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a"
Jul 02 13:36:04.487765 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.487745    2564 scope.go:117] "RemoveContainer" containerID="5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"
Jul 02 13:36:04.488075 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:36:04.488051    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01\": container with ID starting with 5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01 not found: ID does not exist" containerID="5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"
Jul 02 13:36:04.488144 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.488088    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01"} err="failed to get container status \"5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01\": rpc error: code = NotFound desc = could not find container \"5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01\": container with ID starting with 5e5f8fd5fc320d05bba63cdcd2689890ae762a21a65a78ed3c97930ebf0d8a01 not found: ID does not exist"
Jul 02 13:36:04.488144 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.488112    2564 scope.go:117] "RemoveContainer" containerID="10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a"
Jul 02 13:36:04.488595 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:36:04.488516    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a\": container with ID starting with 10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a not found: ID does not exist" containerID="10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a"
Jul 02 13:36:04.488595 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.488568    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a"} err="failed to get container status \"10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a\": rpc error: code = NotFound desc = could not find container \"10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a\": container with ID starting with 10a7ebfa7b131ad43defabec5305aa89f10adacbcaa504500b915d2e8fa01d1a not found: ID does not exist"
Jul 02 13:36:04.491427 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.491402    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"]
Jul 02 13:36:04.495369 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:04.495325    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74"]
Jul 02 13:36:05.762354 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:05.762317    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" path="/var/lib/kubelet/pods/e5389ab4-790d-47d7-8d73-95d0d8edf24e/volumes"
Jul 02 13:36:06.444554 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:06.444491    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:36:16.444426 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:16.444308    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:36:19.039911 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.039863    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" containerID="cri-o://9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d" gracePeriod=45
Jul 02 13:36:19.660568 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.660505    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 02 13:36:19.811226 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.811191    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:36:19.984727 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984689    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcprp\" (UniqueName: \"kubernetes.io/projected/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kube-api-access-qcprp\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.984942 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984757    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-home\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.984942 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tmp-dir\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.984942 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984816    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-dshm\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.984942 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984854    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-model-cache\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.985157 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984950    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kserve-provision-location\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.985157 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.984975    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tls-certs\") pod \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\" (UID: \"8006303f-68d9-4415-86f4-6cdc3c27e3b8\") "
Jul 02 13:36:19.985157 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.985138    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:19.985323 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.985259    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:19.985794 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.985760    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-home" (OuterVolumeSpecName: "home") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:19.987206 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.987178    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:36:19.987379 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.987357    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-dshm" (OuterVolumeSpecName: "dshm") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:19.987462 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:19.987419    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kube-api-access-qcprp" (OuterVolumeSpecName: "kube-api-access-qcprp") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "kube-api-access-qcprp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:36:20.002651 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.002619    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:20.049450 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.049404    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8006303f-68d9-4415-86f4-6cdc3c27e3b8" (UID: "8006303f-68d9-4415-86f4-6cdc3c27e3b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:20.085757 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.085721    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:20.085757 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.085750    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:20.085757 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.085760    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:20.086063 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.085768    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:20.086063 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.085800    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8006303f-68d9-4415-86f4-6cdc3c27e3b8-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:20.086063 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.085829    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcprp\" (UniqueName: \"kubernetes.io/projected/8006303f-68d9-4415-86f4-6cdc3c27e3b8-kube-api-access-qcprp\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:36:20.570496 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.570447    2564 generic.go:356] "Generic (PLEG): container finished" podID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerID="9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d" exitCode=0
Jul 02 13:36:20.570705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.570510    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"
Jul 02 13:36:20.570705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.570547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" event={"ID":"8006303f-68d9-4415-86f4-6cdc3c27e3b8","Type":"ContainerDied","Data":"9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d"}
Jul 02 13:36:20.570705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.570610    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr" event={"ID":"8006303f-68d9-4415-86f4-6cdc3c27e3b8","Type":"ContainerDied","Data":"e105b40e5c66fbba14d658d7361192da3b4a6b78975b85380f14fb2c20f103a1"}
Jul 02 13:36:20.570705 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.570638    2564 scope.go:117] "RemoveContainer" containerID="9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d"
Jul 02 13:36:20.580100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.580079    2564 scope.go:117] "RemoveContainer" containerID="7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827"
Jul 02 13:36:20.591150 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.591127    2564 scope.go:117] "RemoveContainer" containerID="9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d"
Jul 02 13:36:20.591434 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:36:20.591412    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d\": container with ID starting with 9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d not found: ID does not exist" containerID="9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d"
Jul 02 13:36:20.591511 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.591448    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d"} err="failed to get container status \"9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d\": rpc error: code = NotFound desc = could not find container \"9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d\": container with ID starting with 9db46fc7da22db48358d15b8ddbd8169fd54e395f28afaba4fecf0ff4ec88f8d not found: ID does not exist"
Jul 02 13:36:20.591511 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.591476    2564 scope.go:117] "RemoveContainer" containerID="7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827"
Jul 02 13:36:20.591771 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:36:20.591751    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827\": container with ID starting with 7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827 not found: ID does not exist" containerID="7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827"
Jul 02 13:36:20.591859 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.591781    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827"} err="failed to get container status \"7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827\": rpc error: code = NotFound desc = could not find container \"7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827\": container with ID starting with 7f0db0ba87e680df88ab7be29de1907ef0a372972d36760a8596c0fe90893827 not found: ID does not exist"
Jul 02 13:36:20.600434 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.600410    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"]
Jul 02 13:36:20.605032 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:20.605008    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr"]
Jul 02 13:36:21.767571 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:21.767510    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" path="/var/lib/kubelet/pods/8006303f-68d9-4415-86f4-6cdc3c27e3b8/volumes"
Jul 02 13:36:26.444518 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:26.444475    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:36:36.444756 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:36.444703    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:36:46.444647 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:46.444599    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:36:56.444158 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:36:56.444114    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 02 13:37:05.892854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.892821    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"]
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893164    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="storage-initializer"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893175    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="storage-initializer"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893184    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="storage-initializer"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893189    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="storage-initializer"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893197    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893203    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893213    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="main"
Jul 02 13:37:05.893229 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893218    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="main"
Jul 02 13:37:05.893549 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893268    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5389ab4-790d-47d7-8d73-95d0d8edf24e" containerName="main"
Jul 02 13:37:05.893549 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.893280    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8006303f-68d9-4415-86f4-6cdc3c27e3b8" containerName="main"
Jul 02 13:37:05.898142 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.898121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.901029 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.900997    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 02 13:37:05.909369 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.909346    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"]
Jul 02 13:37:05.987621 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987564    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.987621 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.987849 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.987849 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jhm\" (UniqueName: \"kubernetes.io/projected/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kube-api-access-b8jhm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.987849 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.987849 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:05.987849 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:05.987818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088606 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jhm\" (UniqueName: \"kubernetes.io/projected/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kube-api-access-b8jhm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.088993 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.088810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.089096 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.089076    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.089191 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.089166    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.089266 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.089237    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.089325 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.089306    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.091075 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.091050    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.091385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.091361    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.099164 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.099138    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jhm\" (UniqueName: \"kubernetes.io/projected/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kube-api-access-b8jhm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.209692 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.209606    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:06.367230 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.367202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"]
Jul 02 13:37:06.369991 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:37:06.369960    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40e8708_f2ee_4c10_ad6d_44ef731444ff.slice/crio-4b66462d7a127d39089253bb8ed0d954f68c8261f7d413c2b78766b27e164618 WatchSource:0}: Error finding container 4b66462d7a127d39089253bb8ed0d954f68c8261f7d413c2b78766b27e164618: Status 404 returned error can't find the container with id 4b66462d7a127d39089253bb8ed0d954f68c8261f7d413c2b78766b27e164618
Jul 02 13:37:06.454842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.454813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:37:06.462805 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.462725    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:37:06.828340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.828244    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" event={"ID":"d40e8708-f2ee-4c10-ad6d-44ef731444ff","Type":"ContainerStarted","Data":"9e1861d8bcf22fcdcc34a9ed6ca84f7d7296fcb2350e4a3bee6e81c1d3e2d3d4"}
Jul 02 13:37:06.828340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:06.828308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" event={"ID":"d40e8708-f2ee-4c10-ad6d-44ef731444ff","Type":"ContainerStarted","Data":"4b66462d7a127d39089253bb8ed0d954f68c8261f7d413c2b78766b27e164618"}
Jul 02 13:37:10.853813 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:10.853771    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerID="9e1861d8bcf22fcdcc34a9ed6ca84f7d7296fcb2350e4a3bee6e81c1d3e2d3d4" exitCode=0
Jul 02 13:37:10.854209 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:10.853843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" event={"ID":"d40e8708-f2ee-4c10-ad6d-44ef731444ff","Type":"ContainerDied","Data":"9e1861d8bcf22fcdcc34a9ed6ca84f7d7296fcb2350e4a3bee6e81c1d3e2d3d4"}
Jul 02 13:37:11.862555 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:11.862490    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" event={"ID":"d40e8708-f2ee-4c10-ad6d-44ef731444ff","Type":"ContainerStarted","Data":"d4b88391f68f9e8935963c3f51afc1f9be3e05fa79b426fe0eb1cd11744d4bfb"}
Jul 02 13:37:11.889178 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:11.889126    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podStartSLOduration=3.3991052760000002 podStartE2EDuration="6.889110344s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.490005068s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:37:11.888262167 +0000 UTC m=+1406.619300338" watchObservedRunningTime="2026-07-02 13:37:11.889110344 +0000 UTC m=+1406.620148474"
Jul 02 13:37:12.210230 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:12.210190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:16.210058 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:16.210022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:37:16.212036 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:16.211996    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:37:26.210507 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:26.210458    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:37:36.210721 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:36.210670    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:37:36.796624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:36.796586    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"]
Jul 02 13:37:46.210865 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:46.210779    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:37:51.820008 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:51.819938    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" containerID="cri-o://09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154" gracePeriod=45
Jul 02 13:37:52.507279 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.507250    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:37:52.626447 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626420    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m589\" (UniqueName: \"kubernetes.io/projected/53e9be58-570d-4207-a000-4ce1a62d2537-kube-api-access-7m589\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626462    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-kserve-provision-location\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-home\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-tmp-dir\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626601    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9be58-570d-4207-a000-4ce1a62d2537-tls-certs\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626632    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9be58-570d-4207-a000-4ce1a62d2537-cabundle-cert\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626945 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626675    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-dshm\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.626945 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626701    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-model-cache\") pod \"53e9be58-570d-4207-a000-4ce1a62d2537\" (UID: \"53e9be58-570d-4207-a000-4ce1a62d2537\") "
Jul 02 13:37:52.627044 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.626942    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-home" (OuterVolumeSpecName: "home") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:52.627253 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.627214    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-model-cache" (OuterVolumeSpecName: "model-cache") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:52.627448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.627265    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e9be58-570d-4207-a000-4ce1a62d2537-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:37:52.628920 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.628896    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e9be58-570d-4207-a000-4ce1a62d2537-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:37:52.629119 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.629092    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-dshm" (OuterVolumeSpecName: "dshm") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:52.629327 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.629311    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e9be58-570d-4207-a000-4ce1a62d2537-kube-api-access-7m589" (OuterVolumeSpecName: "kube-api-access-7m589") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "kube-api-access-7m589". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:37:52.644763 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.644726    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:52.688517 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.688476    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53e9be58-570d-4207-a000-4ce1a62d2537" (UID: "53e9be58-570d-4207-a000-4ce1a62d2537"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:52.728340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728297    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728360    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728375    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9be58-570d-4207-a000-4ce1a62d2537-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728387    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9be58-570d-4207-a000-4ce1a62d2537-cabundle-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728399    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728410    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728422    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m589\" (UniqueName: \"kubernetes.io/projected/53e9be58-570d-4207-a000-4ce1a62d2537-kube-api-access-7m589\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:52.728503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:52.728436    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9be58-570d-4207-a000-4ce1a62d2537-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:37:53.103997 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.103894    2564 generic.go:356] "Generic (PLEG): container finished" podID="53e9be58-570d-4207-a000-4ce1a62d2537" containerID="09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154" exitCode=0
Jul 02 13:37:53.103997 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.103967    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"
Jul 02 13:37:53.103997 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.103977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" event={"ID":"53e9be58-570d-4207-a000-4ce1a62d2537","Type":"ContainerDied","Data":"09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154"}
Jul 02 13:37:53.104645 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.104049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" event={"ID":"53e9be58-570d-4207-a000-4ce1a62d2537","Type":"ContainerDied","Data":"c62f8b9ffe1d378e0820f2267f7ccb44c2345ee9e1031a9dae63b19ff8089cb9"}
Jul 02 13:37:53.104645 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.104082    2564 scope.go:117] "RemoveContainer" containerID="09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154"
Jul 02 13:37:53.114231 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.114213    2564 scope.go:117] "RemoveContainer" containerID="bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e"
Jul 02 13:37:53.123421 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.123401    2564 scope.go:117] "RemoveContainer" containerID="09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154"
Jul 02 13:37:53.123764 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:37:53.123745    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154\": container with ID starting with 09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154 not found: ID does not exist" containerID="09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154"
Jul 02 13:37:53.123852 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.123771    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154"} err="failed to get container status \"09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154\": rpc error: code = NotFound desc = could not find container \"09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154\": container with ID starting with 09e8576f462acb2e861983cb2cf311a33ee587f0ff7b6f51dce62db6c160a154 not found: ID does not exist"
Jul 02 13:37:53.123852 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.123789    2564 scope.go:117] "RemoveContainer" containerID="bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e"
Jul 02 13:37:53.124032 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:37:53.124011    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e\": container with ID starting with bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e not found: ID does not exist" containerID="bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e"
Jul 02 13:37:53.124091 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.124037    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e"} err="failed to get container status \"bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e\": rpc error: code = NotFound desc = could not find container \"bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e\": container with ID starting with bef2a0e2e21a789f8620ade35339570ff84cb046f7fea6fa58ec002a616dda0e not found: ID does not exist"
Jul 02 13:37:53.128612 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.128521    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"]
Jul 02 13:37:53.135132 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.135105    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz"]
Jul 02 13:37:53.456649 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.456591    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 02 13:37:53.760815 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:53.760720    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" path="/var/lib/kubelet/pods/53e9be58-570d-4207-a000-4ce1a62d2537/volumes"
Jul 02 13:37:56.210177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:37:56.210135    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:38:02.009293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.009257    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"]
Jul 02 13:38:02.009795 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.009658    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="storage-initializer"
Jul 02 13:38:02.009795 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.009670    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="storage-initializer"
Jul 02 13:38:02.009795 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.009679    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main"
Jul 02 13:38:02.009795 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.009684    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main"
Jul 02 13:38:02.009795 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.009746    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53e9be58-570d-4207-a000-4ce1a62d2537" containerName="main"
Jul 02 13:38:02.014466 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.014443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.017623 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.017596    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:38:02.017746 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.017700    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 02 13:38:02.024893 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.024870    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"]
Jul 02 13:38:02.114099 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114099 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114098    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114211    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5cc\" (UniqueName: \"kubernetes.io/projected/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kube-api-access-vg5cc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.114448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.114337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215078 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215078 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5cc\" (UniqueName: \"kubernetes.io/projected/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kube-api-access-vg5cc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215312 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215574 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215719 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215693    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.215957 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215932    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.216076 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215933    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.216076 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215934    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.216076 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.215994    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.217824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.217802    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.217984 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.217966    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.229663 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.229637    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5cc\" (UniqueName: \"kubernetes.io/projected/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kube-api-access-vg5cc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.325650 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.325559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:02.484804 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:02.484769    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"]
Jul 02 13:38:02.488610 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:38:02.488579    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d92b8a_d259_4fdb_9cc5_1af6094e6502.slice/crio-bceed159689fe89d490d8a5474a1196ffe96034f4d3b8392742eb5642ea74620 WatchSource:0}: Error finding container bceed159689fe89d490d8a5474a1196ffe96034f4d3b8392742eb5642ea74620: Status 404 returned error can't find the container with id bceed159689fe89d490d8a5474a1196ffe96034f4d3b8392742eb5642ea74620
Jul 02 13:38:03.166922 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:03.166884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" event={"ID":"f2d92b8a-d259-4fdb-9cc5-1af6094e6502","Type":"ContainerStarted","Data":"1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0"}
Jul 02 13:38:03.167327 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:03.166940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" event={"ID":"f2d92b8a-d259-4fdb-9cc5-1af6094e6502","Type":"ContainerStarted","Data":"bceed159689fe89d490d8a5474a1196ffe96034f4d3b8392742eb5642ea74620"}
Jul 02 13:38:06.210725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:06.210681    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:38:07.190541 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:07.190499    2564 generic.go:356] "Generic (PLEG): container finished" podID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerID="1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0" exitCode=0
Jul 02 13:38:07.190767 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:07.190586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" event={"ID":"f2d92b8a-d259-4fdb-9cc5-1af6094e6502","Type":"ContainerDied","Data":"1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0"}
Jul 02 13:38:08.198259 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:08.198224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" event={"ID":"f2d92b8a-d259-4fdb-9cc5-1af6094e6502","Type":"ContainerStarted","Data":"833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399"}
Jul 02 13:38:08.224858 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:08.224786    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podStartSLOduration=2.757036484 podStartE2EDuration="7.224764811s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.467728327s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:38:08.221952796 +0000 UTC m=+1462.952990950" watchObservedRunningTime="2026-07-02 13:38:08.224764811 +0000 UTC m=+1462.955802942"
Jul 02 13:38:08.325971 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:08.325930    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:12.326233 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:12.326191    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:38:12.328246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:12.328214    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:38:16.211030 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:16.210989    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:38:22.327032 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:22.326979    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:38:26.210889 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:26.210839    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:38:32.327046 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:32.326984    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:38:36.210491 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:36.210450    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:38:42.326205 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:42.326150    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:38:45.855012 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:45.854980    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:38:45.861430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:45.861399    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:38:46.210707 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:46.210661    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:38:52.326518 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:52.326461    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:38:56.210690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:38:56.210644    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:39:02.327077 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:02.327020    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:39:06.210413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:06.210375    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:39:12.326469 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:12.326365    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:39:16.220645 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:16.220610    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:39:16.228755 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:16.228723    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:39:21.104925 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:21.104881    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"]
Jul 02 13:39:22.326822 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:22.326765    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 02 13:39:32.336739 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:32.336707    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:39:32.344929 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:32.344906    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:39:33.730378 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:33.730343    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"]
Jul 02 13:39:36.129147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.129084    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" containerID="cri-o://d4b88391f68f9e8935963c3f51afc1f9be3e05fa79b426fe0eb1cd11744d4bfb" gracePeriod=45
Jul 02 13:39:36.221443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.221400    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 02 13:39:36.753553 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.753500    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerID="d4b88391f68f9e8935963c3f51afc1f9be3e05fa79b426fe0eb1cd11744d4bfb" exitCode=0
Jul 02 13:39:36.753691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.753568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" event={"ID":"d40e8708-f2ee-4c10-ad6d-44ef731444ff","Type":"ContainerDied","Data":"d4b88391f68f9e8935963c3f51afc1f9be3e05fa79b426fe0eb1cd11744d4bfb"}
Jul 02 13:39:36.882472 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.882446    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:39:36.900856 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.900826    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-dshm\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901002 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.900868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-model-cache\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901002 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.900913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tls-certs\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901002 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.900975    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-home\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.901035    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8jhm\" (UniqueName: \"kubernetes.io/projected/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kube-api-access-b8jhm\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.901066    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tmp-dir\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.901103    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kserve-provision-location\") pod \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\" (UID: \"d40e8708-f2ee-4c10-ad6d-44ef731444ff\") "
Jul 02 13:39:36.901172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.901140    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:36.901394 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.901358    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:36.901833 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.901800    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-home" (OuterVolumeSpecName: "home") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:36.903297 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.903264    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-dshm" (OuterVolumeSpecName: "dshm") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:36.903606 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.903580    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:36.903989 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.903951    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kube-api-access-b8jhm" (OuterVolumeSpecName: "kube-api-access-b8jhm") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "kube-api-access-b8jhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:36.920172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.920137    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:36.964703 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:36.964671    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d40e8708-f2ee-4c10-ad6d-44ef731444ff" (UID: "d40e8708-f2ee-4c10-ad6d-44ef731444ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:37.002624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.002586    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:37.002624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.002623    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:37.002624 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.002633    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:37.002796 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.002642    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jhm\" (UniqueName: \"kubernetes.io/projected/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kube-api-access-b8jhm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:37.002796 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.002683    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:37.002796 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.002695    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d40e8708-f2ee-4c10-ad6d-44ef731444ff-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:37.760806 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.760770    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"
Jul 02 13:39:37.760806 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.760801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx" event={"ID":"d40e8708-f2ee-4c10-ad6d-44ef731444ff","Type":"ContainerDied","Data":"4b66462d7a127d39089253bb8ed0d954f68c8261f7d413c2b78766b27e164618"}
Jul 02 13:39:37.761356 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.760863    2564 scope.go:117] "RemoveContainer" containerID="d4b88391f68f9e8935963c3f51afc1f9be3e05fa79b426fe0eb1cd11744d4bfb"
Jul 02 13:39:37.771719 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.771694    2564 scope.go:117] "RemoveContainer" containerID="9e1861d8bcf22fcdcc34a9ed6ca84f7d7296fcb2350e4a3bee6e81c1d3e2d3d4"
Jul 02 13:39:37.789276 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.789252    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"]
Jul 02 13:39:37.792572 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:37.792552    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx"]
Jul 02 13:39:39.759629 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:39.759589    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" path="/var/lib/kubelet/pods/d40e8708-f2ee-4c10-ad6d-44ef731444ff/volumes"
Jul 02 13:39:48.760451 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:48.760386    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main" containerID="cri-o://833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399" gracePeriod=45
Jul 02 13:39:49.317195 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.317169    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:39:49.413969 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.413883    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-model-cache\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.413969 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.413929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-cabundle-cert\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.413974    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-dshm\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tls-certs\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg5cc\" (UniqueName: \"kubernetes.io/projected/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kube-api-access-vg5cc\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414079    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tmp-dir\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414189 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414168    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kserve-provision-location\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414220    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:49.414455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-home\") pod \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\" (UID: \"f2d92b8a-d259-4fdb-9cc5-1af6094e6502\") "
Jul 02 13:39:49.414455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414323    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:49.414655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414616    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.414655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414638    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-cabundle-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.414655 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414617    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-home" (OuterVolumeSpecName: "home") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:49.414827 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.414807    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:49.416186 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.416159    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-dshm" (OuterVolumeSpecName: "dshm") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:49.416313 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.416225    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kube-api-access-vg5cc" (OuterVolumeSpecName: "kube-api-access-vg5cc") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "kube-api-access-vg5cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:49.416313 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.416271    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:49.485222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.485179    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2d92b8a-d259-4fdb-9cc5-1af6094e6502" (UID: "f2d92b8a-d259-4fdb-9cc5-1af6094e6502"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:49.515889 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.515859    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.515889 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.515887    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.516054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.515899    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5cc\" (UniqueName: \"kubernetes.io/projected/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kube-api-access-vg5cc\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.516054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.515910    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.516054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.515919    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.516054 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.515926    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d92b8a-d259-4fdb-9cc5-1af6094e6502-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:39:49.824778 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.824742    2564 generic.go:356] "Generic (PLEG): container finished" podID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerID="833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399" exitCode=0
Jul 02 13:39:49.825236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.824802    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"
Jul 02 13:39:49.825236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.824804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" event={"ID":"f2d92b8a-d259-4fdb-9cc5-1af6094e6502","Type":"ContainerDied","Data":"833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399"}
Jul 02 13:39:49.825236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.824875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7" event={"ID":"f2d92b8a-d259-4fdb-9cc5-1af6094e6502","Type":"ContainerDied","Data":"bceed159689fe89d490d8a5474a1196ffe96034f4d3b8392742eb5642ea74620"}
Jul 02 13:39:49.825236 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.824909    2564 scope.go:117] "RemoveContainer" containerID="833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399"
Jul 02 13:39:49.834011 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.834001    2564 scope.go:117] "RemoveContainer" containerID="1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0"
Jul 02 13:39:49.842417 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.842400    2564 scope.go:117] "RemoveContainer" containerID="833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399"
Jul 02 13:39:49.842726 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:39:49.842701    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399\": container with ID starting with 833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399 not found: ID does not exist" containerID="833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399"
Jul 02 13:39:49.842828 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.842733    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399"} err="failed to get container status \"833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399\": rpc error: code = NotFound desc = could not find container \"833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399\": container with ID starting with 833ca8dc6c8458a4ef76c796daa44683076f602d5f60920591713a5c4d88e399 not found: ID does not exist"
Jul 02 13:39:49.842828 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.842752    2564 scope.go:117] "RemoveContainer" containerID="1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0"
Jul 02 13:39:49.843007 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:39:49.842990    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0\": container with ID starting with 1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0 not found: ID does not exist" containerID="1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0"
Jul 02 13:39:49.843056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.843013    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0"} err="failed to get container status \"1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0\": rpc error: code = NotFound desc = could not find container \"1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0\": container with ID starting with 1ec86c994c8ffb7f6621a779c1ec5307821724ecb73c223e2da12c2dc2a2fce0 not found: ID does not exist"
Jul 02 13:39:49.846662 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.846633    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"]
Jul 02 13:39:49.850618 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:49.850598    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7"]
Jul 02 13:39:50.018998 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.018964    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"]
Jul 02 13:39:50.019378 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019360    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="storage-initializer"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019381    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="storage-initializer"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019395    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019403    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019414    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019422    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019452    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="storage-initializer"
Jul 02 13:39:50.019473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019460    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="storage-initializer"
Jul 02 13:39:50.019832 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019567    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" containerName="main"
Jul 02 13:39:50.019832 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.019580    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40e8708-f2ee-4c10-ad6d-44ef731444ff" containerName="main"
Jul 02 13:39:50.024496 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.024435    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.027431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.027412    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:39:50.027431 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.027412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:39:50.027645 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.027513    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:39:50.027764 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.027748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:39:50.027834 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.027810    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 02 13:39:50.038616 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.038593    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"]
Jul 02 13:39:50.121184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-home\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.121359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tmp-dir\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.121359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-model-cache\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.121359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-dshm\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.121359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.121503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tls-certs\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.121503 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.121427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4l2\" (UniqueName: \"kubernetes.io/projected/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kube-api-access-jf4l2\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222618 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tmp-dir\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222789 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-model-cache\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222789 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-dshm\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222789 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222789 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tls-certs\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4l2\" (UniqueName: \"kubernetes.io/projected/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kube-api-access-jf4l2\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.222976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.222872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-home\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.223089 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.223041    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tmp-dir\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.223147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.223084    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-model-cache\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.223147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.223118    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.223241 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.223187    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-home\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.225075 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.225054    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-dshm\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.225318 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.225301    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tls-certs\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.232150 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.232126    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4l2\" (UniqueName: \"kubernetes.io/projected/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kube-api-access-jf4l2\") pod \"custom-route-timeout-test-kserve-7788bfd7c8-trxvm\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.334112 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.334028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:39:50.475682 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.475656    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"]
Jul 02 13:39:50.477725 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:39:50.477698    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b65bc1_b995_4e6d_adad_ac4d3d9ffc9a.slice/crio-a7f18b6bf47a0cd6cb6aefeb1993181a94192ad852529a075498f6e4a715c5dc WatchSource:0}: Error finding container a7f18b6bf47a0cd6cb6aefeb1993181a94192ad852529a075498f6e4a715c5dc: Status 404 returned error can't find the container with id a7f18b6bf47a0cd6cb6aefeb1993181a94192ad852529a075498f6e4a715c5dc
Jul 02 13:39:50.832830 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.832790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" event={"ID":"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a","Type":"ContainerStarted","Data":"edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5"}
Jul 02 13:39:50.833328 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:50.832855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" event={"ID":"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a","Type":"ContainerStarted","Data":"a7f18b6bf47a0cd6cb6aefeb1993181a94192ad852529a075498f6e4a715c5dc"}
Jul 02 13:39:51.761144 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:51.761103    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d92b8a-d259-4fdb-9cc5-1af6094e6502" path="/var/lib/kubelet/pods/f2d92b8a-d259-4fdb-9cc5-1af6094e6502/volumes"
Jul 02 13:39:54.855227 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:54.855179    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerID="edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5" exitCode=0
Jul 02 13:39:54.855632 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:54.855255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" event={"ID":"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a","Type":"ContainerDied","Data":"edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5"}
Jul 02 13:39:55.862986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:55.862945    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" event={"ID":"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a","Type":"ContainerStarted","Data":"6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9"}
Jul 02 13:39:55.892126 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:55.892066    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podStartSLOduration=2.81280878 podStartE2EDuration="6.892050264s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.079241484s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:55.888404498 +0000 UTC m=+1570.619442643" watchObservedRunningTime="2026-07-02 13:39:55.892050264 +0000 UTC m=+1570.623088440"
Jul 02 13:39:56.334148 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:39:56.334102    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:40:00.334824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:00.334777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:40:00.336345 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:00.336316    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:40:10.334871 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:10.334826    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:40:17.190001 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.189966    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"]
Jul 02 13:40:17.193324 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.193307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.195989 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.195964    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 02 13:40:17.208673 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.208646    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"]
Jul 02 13:40:17.367566 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.367503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.367566 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.367567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.367784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.367674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.367784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.367737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.367853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.367811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.367853 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.367833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz9d\" (UniqueName: \"kubernetes.io/projected/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-kube-api-access-hcz9d\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.468902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469003 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.468996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.469111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.469428    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469555 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.469482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz9d\" (UniqueName: \"kubernetes.io/projected/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-kube-api-access-hcz9d\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469620 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.469574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.469620 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.469607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.470041 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.470019    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.470292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.470271    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.471933 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.471882    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.472100 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.472032    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:17.478342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:17.478313    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz9d\" (UniqueName: \"kubernetes.io/projected/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-kube-api-access-hcz9d\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:20.335546 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:20.335424    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:40:27.257551 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.257491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.263787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.263759    2564 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.358812 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.358775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.358997 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.358945    2564 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.362640 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.362620    2564 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") DevicePath \"csi-ebcc659efedf0f0cc5af941628d822974ce376ed8d2a75e4f8504120510641f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.460686 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.460642    2564 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/86c6b61c91c4d06e5938fc2dd8dea7e3e98bb85489e27e6fc0857c958e8ae788/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.478551 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.478495    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.703791 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.703744    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:27.850511 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:27.850480    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"]
Jul 02 13:40:27.852953 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:40:27.852899    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5c23b3_a8ce_4938_a89d_4ad0341313a4.slice/crio-7ddae81f82a0b4dd7eb801da5e4cde023d3810753808d815d9998cda87f883c7 WatchSource:0}: Error finding container 7ddae81f82a0b4dd7eb801da5e4cde023d3810753808d815d9998cda87f883c7: Status 404 returned error can't find the container with id 7ddae81f82a0b4dd7eb801da5e4cde023d3810753808d815d9998cda87f883c7
Jul 02 13:40:28.036744 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:28.036638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" event={"ID":"2e5c23b3-a8ce-4938-a89d-4ad0341313a4","Type":"ContainerStarted","Data":"85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb"}
Jul 02 13:40:28.036744 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:28.036699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" event={"ID":"2e5c23b3-a8ce-4938-a89d-4ad0341313a4","Type":"ContainerStarted","Data":"7ddae81f82a0b4dd7eb801da5e4cde023d3810753808d815d9998cda87f883c7"}
Jul 02 13:40:28.056997 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:28.056921    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podStartSLOduration=11.056896457 podStartE2EDuration="11.056896457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:28.054673781 +0000 UTC m=+1602.785711928" watchObservedRunningTime="2026-07-02 13:40:28.056896457 +0000 UTC m=+1602.787934589"
Jul 02 13:40:28.704766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:28.704729    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:30.335077 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:30.335026    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:40:37.704430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:37.704382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:40:37.706331 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:37.706302    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:40:40.334798 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:40.334732    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:40:47.704687 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:47.704627    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:40:50.335546 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:50.335487    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:40:57.704847 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:40:57.704788    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:41:00.334717 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:00.334668    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:41:07.704777 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:07.704718    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:41:10.334996 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:10.334941    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:41:17.704888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:17.704829    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:41:20.335254 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:20.335204    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:41:27.705069 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:27.705018    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:41:30.335009 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:30.334961    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:41:37.704767 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:37.704710    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:41:40.335278 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:40.335217    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:41:47.704271 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:47.704216    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:41:50.334514 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:50.334462    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:41:57.704768 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:41:57.704712    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:42:00.344718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:00.344680    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:42:00.353182 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:00.353153    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:42:07.704842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:07.704786    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:42:08.463618 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:08.463576    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"]
Jul 02 13:42:17.705199 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:17.705148    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:42:23.488942 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:23.488896    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" containerID="cri-o://6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9" gracePeriod=45
Jul 02 13:42:24.346182 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.346136    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 02 13:42:24.549971 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.549943    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:42:24.645771 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tmp-dir\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.645771 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645775    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-model-cache\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.646061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-dshm\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.646061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645842    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tls-certs\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.646061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf4l2\" (UniqueName: \"kubernetes.io/projected/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kube-api-access-jf4l2\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.646061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645897    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kserve-provision-location\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.646061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.645915    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-home\") pod \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\" (UID: \"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a\") "
Jul 02 13:42:24.646061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.646038    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:24.646380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.646180    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.646755 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.646728    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-home" (OuterVolumeSpecName: "home") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:24.648669 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.648631    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:42:24.648669 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.648655    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kube-api-access-jf4l2" (OuterVolumeSpecName: "kube-api-access-jf4l2") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "kube-api-access-jf4l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:42:24.648847 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.648738    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-dshm" (OuterVolumeSpecName: "dshm") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:24.658988 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.658946    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:24.712690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.712645    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" (UID: "f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:24.722225 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.722191    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerID="6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9" exitCode=0
Jul 02 13:42:24.722424 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.722255    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"
Jul 02 13:42:24.722424 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.722274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" event={"ID":"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a","Type":"ContainerDied","Data":"6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9"}
Jul 02 13:42:24.722424 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.722339    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm" event={"ID":"f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a","Type":"ContainerDied","Data":"a7f18b6bf47a0cd6cb6aefeb1993181a94192ad852529a075498f6e4a715c5dc"}
Jul 02 13:42:24.722424 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.722362    2564 scope.go:117] "RemoveContainer" containerID="6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9"
Jul 02 13:42:24.735090 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.735065    2564 scope.go:117] "RemoveContainer" containerID="edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5"
Jul 02 13:42:24.743823 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.743800    2564 scope.go:117] "RemoveContainer" containerID="6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9"
Jul 02 13:42:24.744094 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:42:24.744074    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9\": container with ID starting with 6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9 not found: ID does not exist" containerID="6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9"
Jul 02 13:42:24.744177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.744109    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9"} err="failed to get container status \"6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9\": rpc error: code = NotFound desc = could not find container \"6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9\": container with ID starting with 6b92545b714d7115313cc58985bd0165a6bffc17016201a4edc3f6d719e370d9 not found: ID does not exist"
Jul 02 13:42:24.744177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.744134    2564 scope.go:117] "RemoveContainer" containerID="edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5"
Jul 02 13:42:24.744412 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:42:24.744388    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5\": container with ID starting with edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5 not found: ID does not exist" containerID="edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5"
Jul 02 13:42:24.744452 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.744428    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5"} err="failed to get container status \"edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5\": rpc error: code = NotFound desc = could not find container \"edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5\": container with ID starting with edb7a4020e4a06f2ebef8f531b20e379a1c8e6227740a74b3902c423677ebea5 not found: ID does not exist"
Jul 02 13:42:24.746893 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.746871    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.746893 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.746892    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.747031 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.746900    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.747031 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.746909    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4l2\" (UniqueName: \"kubernetes.io/projected/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kube-api-access-jf4l2\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.747031 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.746917    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.747031 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.746927    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:42:24.748161 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.748141    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"]
Jul 02 13:42:24.751147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:24.751127    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm"]
Jul 02 13:42:25.761547 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:25.761497    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" path="/var/lib/kubelet/pods/f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a/volumes"
Jul 02 13:42:27.704659 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:27.704603    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 02 13:42:34.994757 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:34.994714    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"]
Jul 02 13:42:34.995280 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:34.995257    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main"
Jul 02 13:42:34.995332 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:34.995287    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main"
Jul 02 13:42:34.995332 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:34.995316    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="storage-initializer"
Jul 02 13:42:34.995332 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:34.995325    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="storage-initializer"
Jul 02 13:42:34.995446 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:34.995434    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b65bc1-b995-4e6d-adad-ac4d3d9ffc9a" containerName="main"
Jul 02 13:42:35.001952 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.001931    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.004934 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.004901    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:42:35.005241 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.005219    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 02 13:42:35.007342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.007322    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"]
Jul 02 13:42:35.031792 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.031751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-tmp-dir\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.031980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.031802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-kserve-provision-location\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.031980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.031852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-home\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.031980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.031885    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-model-cache\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.032166 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.032004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43834f1c-90be-4701-9a49-27a5075ae79b-tls-certs\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.032166 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.032062    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-dshm\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.032166 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.032106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgl4\" (UniqueName: \"kubernetes.io/projected/43834f1c-90be-4701-9a49-27a5075ae79b-kube-api-access-vwgl4\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133485 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-tmp-dir\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-kserve-provision-location\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-home\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-model-cache\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43834f1c-90be-4701-9a49-27a5075ae79b-tls-certs\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133698 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-dshm\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.133984 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgl4\" (UniqueName: \"kubernetes.io/projected/43834f1c-90be-4701-9a49-27a5075ae79b-kube-api-access-vwgl4\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.134045 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.133987    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-kserve-provision-location\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.134045 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.134012    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-tmp-dir\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.134405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.134382    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-home\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.134405 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.134391    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-model-cache\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.136339 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.136308    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43834f1c-90be-4701-9a49-27a5075ae79b-tls-certs\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.136422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.136354    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-dshm\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.142758 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.142732    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgl4\" (UniqueName: \"kubernetes.io/projected/43834f1c-90be-4701-9a49-27a5075ae79b-kube-api-access-vwgl4\") pod \"router-with-refs-test-kserve-78d588d54b-cbmnc\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.313227 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.313118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:35.455589 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.455560    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"]
Jul 02 13:42:35.457728 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:42:35.457696    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43834f1c_90be_4701_9a49_27a5075ae79b.slice/crio-b5f4b842951f822d60ac0c25b47aefea221179884a7682c1dce6bf2f737ca483 WatchSource:0}: Error finding container b5f4b842951f822d60ac0c25b47aefea221179884a7682c1dce6bf2f737ca483: Status 404 returned error can't find the container with id b5f4b842951f822d60ac0c25b47aefea221179884a7682c1dce6bf2f737ca483
Jul 02 13:42:35.788139 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.788084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" event={"ID":"43834f1c-90be-4701-9a49-27a5075ae79b","Type":"ContainerStarted","Data":"2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e"}
Jul 02 13:42:35.788354 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:35.788163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" event={"ID":"43834f1c-90be-4701-9a49-27a5075ae79b","Type":"ContainerStarted","Data":"b5f4b842951f822d60ac0c25b47aefea221179884a7682c1dce6bf2f737ca483"}
Jul 02 13:42:37.715184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:37.715152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:42:37.729238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:37.729207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:42:40.817105 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:40.817066    2564 generic.go:356] "Generic (PLEG): container finished" podID="43834f1c-90be-4701-9a49-27a5075ae79b" containerID="2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e" exitCode=0
Jul 02 13:42:40.817609 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:40.817145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" event={"ID":"43834f1c-90be-4701-9a49-27a5075ae79b","Type":"ContainerDied","Data":"2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e"}
Jul 02 13:42:41.824723 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:41.824685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" event={"ID":"43834f1c-90be-4701-9a49-27a5075ae79b","Type":"ContainerStarted","Data":"d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e"}
Jul 02 13:42:41.846810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:41.846755    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podStartSLOduration=2.703462751 podStartE2EDuration="7.846738892s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.143276141s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:42:41.845148364 +0000 UTC m=+1736.576186519" watchObservedRunningTime="2026-07-02 13:42:41.846738892 +0000 UTC m=+1736.577777022"
Jul 02 13:42:42.313720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:42.313680    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:45.314127 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:45.314086    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:42:45.315810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:45.315771    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:42:55.314287 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:55.314236    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:42:55.980903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:42:55.980865    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"]
Jul 02 13:43:05.313841 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:05.313795    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:43:11.008004 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.007936    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" containerID="cri-o://85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb" gracePeriod=45
Jul 02 13:43:11.787355 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.787285    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:43:11.879598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879555    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-model-cache\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.879765 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879707    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.879827 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879809    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcz9d\" (UniqueName: \"kubernetes.io/projected/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-kube-api-access-hcz9d\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.879872 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879841    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-dshm\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.879872 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879857    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:43:11.879976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tls-certs\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.879976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879920    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tmp-dir\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.879976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.879942    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-home\") pod \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\" (UID: \"2e5c23b3-a8ce-4938-a89d-4ad0341313a4\") "
Jul 02 13:43:11.880225 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.880203    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:43:11.880933 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.880901    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-home" (OuterVolumeSpecName: "home") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:43:11.882203 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.882179    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-dshm" (OuterVolumeSpecName: "dshm") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:43:11.882583 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.882519    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:43:11.882682 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.882585    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-kube-api-access-hcz9d" (OuterVolumeSpecName: "kube-api-access-hcz9d") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "kube-api-access-hcz9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:43:11.883976 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.883954    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7" (OuterVolumeSpecName: "kserve-pvc-source") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "pvc-44641bc5-8148-4920-9d29-b75c22fd719a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 02 13:43:11.896851 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.896814    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e5c23b3-a8ce-4938-a89d-4ad0341313a4" (UID: "2e5c23b3-a8ce-4938-a89d-4ad0341313a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:43:11.981061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.981010    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:43:11.981061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.981063    2564 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") on node \"ip-10-0-138-135.ec2.internal\" "
Jul 02 13:43:11.981335 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.981076    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcz9d\" (UniqueName: \"kubernetes.io/projected/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-kube-api-access-hcz9d\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:43:11.981335 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.981085    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:43:11.981335 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.981095    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:43:11.981335 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.981103    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c23b3-a8ce-4938-a89d-4ad0341313a4-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:43:11.989959 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.989928    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerID="85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb" exitCode=0
Jul 02 13:43:11.990134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.989985    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"
Jul 02 13:43:11.990134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.990003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" event={"ID":"2e5c23b3-a8ce-4938-a89d-4ad0341313a4","Type":"ContainerDied","Data":"85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb"}
Jul 02 13:43:11.990134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.990076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" event={"ID":"2e5c23b3-a8ce-4938-a89d-4ad0341313a4","Type":"ContainerDied","Data":"7ddae81f82a0b4dd7eb801da5e4cde023d3810753808d815d9998cda87f883c7"}
Jul 02 13:43:11.990134 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.990097    2564 scope.go:117] "RemoveContainer" containerID="85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb"
Jul 02 13:43:11.999703 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:11.999685    2564 scope.go:117] "RemoveContainer" containerID="85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb"
Jul 02 13:43:12.000002 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:43:11.999984    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb\": container with ID starting with 85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb not found: ID does not exist" containerID="85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb"
Jul 02 13:43:12.000065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:12.000009    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb"} err="failed to get container status \"85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb\": rpc error: code = NotFound desc = could not find container \"85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb\": container with ID starting with 85e5bac8fbb742d17e0d7921f196a2b0b33fb56394c278e6535b6df5bf81daeb not found: ID does not exist"
Jul 02 13:43:12.013273 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:12.013254    2564 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7") on node "ip-10-0-138-135.ec2.internal"
Jul 02 13:43:12.014740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:12.014712    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"]
Jul 02 13:43:12.018945 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:12.018915    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc"]
Jul 02 13:43:12.082101 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:12.082002    2564 reconciler_common.go:299] "Volume detached for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"csi-ebcc659efedf0f0cc5af941628d822974ce376ed8d2a75e4f8504120510641f9\""
Jul 02 13:43:12.716400 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:12.716340    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 02 13:43:13.761916 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:13.761868    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" path="/var/lib/kubelet/pods/2e5c23b3-a8ce-4938-a89d-4ad0341313a4/volumes"
Jul 02 13:43:15.313571 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:15.313500    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:43:25.314213 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:25.314164    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:43:26.824656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.824621    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"]
Jul 02 13:43:26.825239 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.825148    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main"
Jul 02 13:43:26.825239 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.825171    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main"
Jul 02 13:43:26.825381 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.825261    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5c23b3-a8ce-4938-a89d-4ad0341313a4" containerName="main"
Jul 02 13:43:26.828800 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.828774    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:26.831444 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.831413    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-ss8xm\""
Jul 02 13:43:26.831778 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.831754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 02 13:43:26.840895 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.840869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"]
Jul 02 13:43:26.849691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.849659    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"]
Jul 02 13:43:26.853284 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.853265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:26.864006 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.863977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"]
Jul 02 13:43:26.912704 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912667    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:26.912863 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:26.912863 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:26.912863 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912795    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a61ddaa-6ef6-400a-925b-80a204a97ab0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:26.912863 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:26.913097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912885    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:26.913097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:26.913097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.912951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:26.913097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.913010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:26.913097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.913060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d6d966-062a-4c8e-8461-cf08c57dddad-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:26.913097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:26.913085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfxg\" (UniqueName: \"kubernetes.io/projected/94d6d966-062a-4c8e-8461-cf08c57dddad-kube-api-access-9qfxg\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.013974 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.013927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.013974 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.013973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2l6s\" (UniqueName: \"kubernetes.io/projected/9a61ddaa-6ef6-400a-925b-80a204a97ab0-kube-api-access-n2l6s\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.014238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d6d966-062a-4c8e-8461-cf08c57dddad-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.014238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfxg\" (UniqueName: \"kubernetes.io/projected/94d6d966-062a-4c8e-8461-cf08c57dddad-kube-api-access-9qfxg\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.014238 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.014590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.014590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a61ddaa-6ef6-400a-925b-80a204a97ab0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014471    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014498    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014969 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014666    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.014969 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014779    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.014969 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.014876    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.015141 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.015075    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.016993 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.016969    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d6d966-062a-4c8e-8461-cf08c57dddad-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.017246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.017224    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a61ddaa-6ef6-400a-925b-80a204a97ab0-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.017299 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.017245    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a61ddaa-6ef6-400a-925b-80a204a97ab0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.017359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.017337    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d6d966-062a-4c8e-8461-cf08c57dddad-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.023481 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.023453    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfxg\" (UniqueName: \"kubernetes.io/projected/94d6d966-062a-4c8e-8461-cf08c57dddad-kube-api-access-9qfxg\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" (UID: \"94d6d966-062a-4c8e-8461-cf08c57dddad\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:43:27.115604 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.115485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2l6s\" (UniqueName: \"kubernetes.io/projected/9a61ddaa-6ef6-400a-925b-80a204a97ab0-kube-api-access-n2l6s\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:27.124602 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:27.124573    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2l6s\" (UniqueName: \"kubernetes.io/projected/9a61ddaa-6ef6-400a-925b-80a204a97ab0-kube-api-access-n2l6s\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" (UID: \"9a61ddaa-6ef6-400a-925b-80a204a97ab0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:43:35.314112 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:35.314061    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:43:45.313453 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:45.313406    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:43:45.892016 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:45.891984    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:43:45.899443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:45.899421    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:43:55.313895 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:43:55.313841    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:44:05.313872 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:44:05.313825    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:44:15.313876 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:44:15.313826    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:44:25.314082 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:44:25.314035    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:44:35.323710 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:44:35.323676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:44:35.331808 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:44:35.331785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:44:46.713199 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:44:46.713164    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"]
Jul 02 13:45:01.739218 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:01.739169    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" containerID="cri-o://d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e" gracePeriod=45
Jul 02 13:45:02.325074 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.325036    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 02 13:45:02.494712 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.494687    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:45:02.600718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600635    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwgl4\" (UniqueName: \"kubernetes.io/projected/43834f1c-90be-4701-9a49-27a5075ae79b-kube-api-access-vwgl4\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.600718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-tmp-dir\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.600938 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600739    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-home\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.600938 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-model-cache\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.600938 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600819    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43834f1c-90be-4701-9a49-27a5075ae79b-tls-certs\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.600938 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-dshm\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.600938 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.600899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-kserve-provision-location\") pod \"43834f1c-90be-4701-9a49-27a5075ae79b\" (UID: \"43834f1c-90be-4701-9a49-27a5075ae79b\") "
Jul 02 13:45:02.601190 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.601097    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-model-cache" (OuterVolumeSpecName: "model-cache") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:45:02.601295 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.601249    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.601588 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.601563    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-home" (OuterVolumeSpecName: "home") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:45:02.602949 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.602896    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43834f1c-90be-4701-9a49-27a5075ae79b-kube-api-access-vwgl4" (OuterVolumeSpecName: "kube-api-access-vwgl4") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "kube-api-access-vwgl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:45:02.603066 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.603001    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43834f1c-90be-4701-9a49-27a5075ae79b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:45:02.603323 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.603292    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-dshm" (OuterVolumeSpecName: "dshm") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:45:02.611688 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.611659    2564 generic.go:356] "Generic (PLEG): container finished" podID="43834f1c-90be-4701-9a49-27a5075ae79b" containerID="d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e" exitCode=0
Jul 02 13:45:02.611803 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.611713    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"
Jul 02 13:45:02.611803 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.611735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" event={"ID":"43834f1c-90be-4701-9a49-27a5075ae79b","Type":"ContainerDied","Data":"d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e"}
Jul 02 13:45:02.611912 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.611813    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc" event={"ID":"43834f1c-90be-4701-9a49-27a5075ae79b","Type":"ContainerDied","Data":"b5f4b842951f822d60ac0c25b47aefea221179884a7682c1dce6bf2f737ca483"}
Jul 02 13:45:02.611912 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.611844    2564 scope.go:117] "RemoveContainer" containerID="d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e"
Jul 02 13:45:02.616425 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.616392    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:45:02.621810 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.621788    2564 scope.go:117] "RemoveContainer" containerID="2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e"
Jul 02 13:45:02.629986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.629966    2564 scope.go:117] "RemoveContainer" containerID="d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e"
Jul 02 13:45:02.630295 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:45:02.630276    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e\": container with ID starting with d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e not found: ID does not exist" containerID="d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e"
Jul 02 13:45:02.630357 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.630304    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e"} err="failed to get container status \"d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e\": rpc error: code = NotFound desc = could not find container \"d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e\": container with ID starting with d33813632d4db5f50dfb3109a6be26ffdc43ac33d7574a698ec7618788609b1e not found: ID does not exist"
Jul 02 13:45:02.630357 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.630322    2564 scope.go:117] "RemoveContainer" containerID="2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e"
Jul 02 13:45:02.630613 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:45:02.630592    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e\": container with ID starting with 2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e not found: ID does not exist" containerID="2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e"
Jul 02 13:45:02.630699 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.630620    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e"} err="failed to get container status \"2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e\": rpc error: code = NotFound desc = could not find container \"2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e\": container with ID starting with 2e3a01e27d54d8afdd3f18cd06d1f5ed6d19df5847fd183e9983a6460e77968e not found: ID does not exist"
Jul 02 13:45:02.674460 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.674411    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43834f1c-90be-4701-9a49-27a5075ae79b" (UID: "43834f1c-90be-4701-9a49-27a5075ae79b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:45:02.701883 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.701859    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43834f1c-90be-4701-9a49-27a5075ae79b-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.701883 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.701882    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.701992 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.701891    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.701992 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.701900    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwgl4\" (UniqueName: \"kubernetes.io/projected/43834f1c-90be-4701-9a49-27a5075ae79b-kube-api-access-vwgl4\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.701992 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.701909    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.701992 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.701918    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43834f1c-90be-4701-9a49-27a5075ae79b-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:45:02.935030 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.935002    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"]
Jul 02 13:45:02.939249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:02.939227    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc"]
Jul 02 13:45:03.760243 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:03.760206    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" path="/var/lib/kubelet/pods/43834f1c-90be-4701-9a49-27a5075ae79b/volumes"
Jul 02 13:45:16.052282 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.052186    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"]
Jul 02 13:45:16.052819 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.052624    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="storage-initializer"
Jul 02 13:45:16.052819 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.052639    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="storage-initializer"
Jul 02 13:45:16.052819 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.052653    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main"
Jul 02 13:45:16.052819 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.052659    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main"
Jul 02 13:45:16.052819 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.052723    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43834f1c-90be-4701-9a49-27a5075ae79b" containerName="main"
Jul 02 13:45:16.057675 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.057653    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.060360 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.060340    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-7ln8r\""
Jul 02 13:45:16.060469 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.060340    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 02 13:45:16.060469 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.060364    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:45:16.064128 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.064100    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"]
Jul 02 13:45:16.067772 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.067752    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.069961 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.069941    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"]
Jul 02 13:45:16.079557 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.079517    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"]
Jul 02 13:45:16.122426 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122643 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122643 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.122643 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vrx\" (UniqueName: \"kubernetes.io/projected/1732cd44-856d-4dee-8588-b29c4d4fa059-kube-api-access-94vrx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.122643 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cd44-856d-4dee-8588-b29c4d4fa059-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.122643 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdr7\" (UniqueName: \"kubernetes.io/projected/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kube-api-access-4kdr7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.122900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.122866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.223317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.223521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vrx\" (UniqueName: \"kubernetes.io/projected/1732cd44-856d-4dee-8588-b29c4d4fa059-kube-api-access-94vrx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cd44-856d-4dee-8588-b29c4d4fa059-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdr7\" (UniqueName: \"kubernetes.io/projected/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kube-api-access-4kdr7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.223718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223718 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223621    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.223877 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223730    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.223877 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223751    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223877 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223877 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.223877 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.224140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.224140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223850    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.224140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.224140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223929    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.224140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.223983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.224140 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.224082    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.224448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.224214    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.224448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.224232    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.224448 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.224281    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.225986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.225960    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.226115 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.226067    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.226233 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.226215    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cd44-856d-4dee-8588-b29c4d4fa059-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.226381 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.226362    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.231459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.231433    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdr7\" (UniqueName: \"kubernetes.io/projected/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kube-api-access-4kdr7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.231781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.231762    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vrx\" (UniqueName: \"kubernetes.io/projected/1732cd44-856d-4dee-8588-b29c4d4fa059-kube-api-access-94vrx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.367827 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.367747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:16.378165 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.378144    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:16.522045 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.522008    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"]
Jul 02 13:45:16.522334 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:45:16.522314    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1732cd44_856d_4dee_8588_b29c4d4fa059.slice/crio-ec3f88f5680795aabd9941468a99d13d8bdc49578a33346b1482d5031278aa9c WatchSource:0}: Error finding container ec3f88f5680795aabd9941468a99d13d8bdc49578a33346b1482d5031278aa9c: Status 404 returned error can't find the container with id ec3f88f5680795aabd9941468a99d13d8bdc49578a33346b1482d5031278aa9c
Jul 02 13:45:16.524380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.524360    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:45:16.544802 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.544779    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"]
Jul 02 13:45:16.546195 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:45:16.546170    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36d3511_80e7_40e5_9839_d61bd5d08fdf.slice/crio-d744297e2f16f23fa07dcfa22d83bed2af5ff0093cc43c1bdcafcd9c8ccbf390 WatchSource:0}: Error finding container d744297e2f16f23fa07dcfa22d83bed2af5ff0093cc43c1bdcafcd9c8ccbf390: Status 404 returned error can't find the container with id d744297e2f16f23fa07dcfa22d83bed2af5ff0093cc43c1bdcafcd9c8ccbf390
Jul 02 13:45:16.683882 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.683850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" event={"ID":"c36d3511-80e7-40e5-9839-d61bd5d08fdf","Type":"ContainerStarted","Data":"9ae82e3b37a66df889b124de204b654ae9c92210b5200fed7baf59676016913a"}
Jul 02 13:45:16.684065 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.683907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" event={"ID":"c36d3511-80e7-40e5-9839-d61bd5d08fdf","Type":"ContainerStarted","Data":"d744297e2f16f23fa07dcfa22d83bed2af5ff0093cc43c1bdcafcd9c8ccbf390"}
Jul 02 13:45:16.685188 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:16.685156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerStarted","Data":"ec3f88f5680795aabd9941468a99d13d8bdc49578a33346b1482d5031278aa9c"}
Jul 02 13:45:17.692362 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:17.692325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerStarted","Data":"c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495"}
Jul 02 13:45:17.693030 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:17.692623    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:18.705741 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:18.705695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerStarted","Data":"7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e"}
Jul 02 13:45:20.719640 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:20.719609    2564 generic.go:356] "Generic (PLEG): container finished" podID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerID="9ae82e3b37a66df889b124de204b654ae9c92210b5200fed7baf59676016913a" exitCode=0
Jul 02 13:45:20.720107 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:20.719680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" event={"ID":"c36d3511-80e7-40e5-9839-d61bd5d08fdf","Type":"ContainerDied","Data":"9ae82e3b37a66df889b124de204b654ae9c92210b5200fed7baf59676016913a"}
Jul 02 13:45:21.734011 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:21.733977    2564 generic.go:356] "Generic (PLEG): container finished" podID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerID="7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e" exitCode=0
Jul 02 13:45:21.734473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:21.734040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerDied","Data":"7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e"}
Jul 02 13:45:21.736586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:21.736561    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" event={"ID":"c36d3511-80e7-40e5-9839-d61bd5d08fdf","Type":"ContainerStarted","Data":"ccb74d9837d87432151e13021e0aee920d667f8fceb93265644422b93db12ce7"}
Jul 02 13:45:21.779408 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:21.779361    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podStartSLOduration=1.739053169 podStartE2EDuration="5.779341668s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.040288499s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:21.777074836 +0000 UTC m=+1896.508112982" watchObservedRunningTime="2026-07-02 13:45:21.779341668 +0000 UTC m=+1896.510379803"
Jul 02 13:45:22.745292 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:22.745250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerStarted","Data":"cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded"}
Jul 02 13:45:22.769945 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:22.769840    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podStartSLOduration=2.562996563 podStartE2EDuration="6.769817764s" totalImagesPullingTime="784.840333ms" totalInitContainerRuntime="3.421980868s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:22.768187127 +0000 UTC m=+1897.499225294" watchObservedRunningTime="2026-07-02 13:45:22.769817764 +0000 UTC m=+1897.500855894"
Jul 02 13:45:26.368346 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:26.368313    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:26.368872 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:26.368411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:26.369937 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:26.369909    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:45:26.379174 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:26.379152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:26.379288 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:26.379202    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:45:26.380726 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:26.380687    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:45:29.843441 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:45:29.843400    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:45:29.865585 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:45:29.865547    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:45:30.795978 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:30.795946    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:45:30.796155 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:30.795946    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:45:36.368686 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:36.368630    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:45:36.379487 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:36.379447    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:45:36.386480 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:36.386455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:45:46.368425 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:46.368364    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:45:46.378977 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:46.378593    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:45:56.368455 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:56.368242    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:45:56.378595 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:45:56.378548    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:46:06.369080 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:06.368942    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:46:06.379521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:06.379489    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:46:16.368759 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:16.368709    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:46:16.380618 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:16.380578    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:46:26.369001 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:26.368951    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:46:26.379037 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:26.378999    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:46:36.368457 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:36.368412    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:46:36.379514 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:36.379476    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:46:46.368578 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:46.368466    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:46:46.379838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:46.379803    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:46:56.372659 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:56.372505    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:46:56.378614 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:46:56.378578    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:47:06.368734 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:06.368687    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:47:06.379356 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:06.379326    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:47:16.368211 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:16.368164    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:47:16.379217 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:16.379184    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:47:26.368376 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:26.368326    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:47:26.378613 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:26.378577    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:47:33.796570 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:47:33.796512    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:47:33.797020 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:47:33.796614    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:47:34.487570 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:34.487511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:47:34.487742 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:34.487541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:47:36.369123 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:36.369080    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:47:36.378629 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:36.378601    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:47:46.369177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:46.369130    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8001/health\": dial tcp 10.132.0.56:8001: connect: connection refused"
Jul 02 13:47:46.379575 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:46.379540    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 02 13:47:56.377835 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:56.377801    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:47:56.388996 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:56.388970    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:47:56.390673 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:56.390651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:47:56.397096 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:47:56.397074    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:48:24.945484 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:24.945445    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"]
Jul 02 13:48:24.955719 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:24.955648    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"]
Jul 02 13:48:39.971939 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:39.971903    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main" containerID="cri-o://ccb74d9837d87432151e13021e0aee920d667f8fceb93265644422b93db12ce7" gracePeriod=45
Jul 02 13:48:39.986607 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:39.986562    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main" containerID="cri-o://cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded" gracePeriod=45
Jul 02 13:48:40.719891 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.719841    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="llm-d-routing-sidecar" containerID="cri-o://c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495" gracePeriod=45
Jul 02 13:48:40.834397 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.834365    2564 generic.go:356] "Generic (PLEG): container finished" podID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerID="cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded" exitCode=0
Jul 02 13:48:40.834567 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.834454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerDied","Data":"cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded"}
Jul 02 13:48:40.836757 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.836729    2564 generic.go:356] "Generic (PLEG): container finished" podID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerID="ccb74d9837d87432151e13021e0aee920d667f8fceb93265644422b93db12ce7" exitCode=0
Jul 02 13:48:40.836880 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.836770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" event={"ID":"c36d3511-80e7-40e5-9839-d61bd5d08fdf","Type":"ContainerDied","Data":"ccb74d9837d87432151e13021e0aee920d667f8fceb93265644422b93db12ce7"}
Jul 02 13:48:40.979867 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.979843    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:48:40.989429 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:40.989406    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:48:41.035408 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tmp-dir\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035439    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kdr7\" (UniqueName: \"kubernetes.io/projected/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kube-api-access-4kdr7\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-model-cache\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035505    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94vrx\" (UniqueName: \"kubernetes.io/projected/1732cd44-856d-4dee-8588-b29c4d4fa059-kube-api-access-94vrx\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.035601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-dshm\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035601 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035593    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kserve-provision-location\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035619    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-kserve-provision-location\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-model-cache\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035696    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tls-certs\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035728    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cd44-856d-4dee-8588-b29c4d4fa059-tls-certs\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035754    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-home\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035778    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-home\") pod \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\" (UID: \"c36d3511-80e7-40e5-9839-d61bd5d08fdf\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035836    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-dshm\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.035888 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035881    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-tmp-dir\") pod \"1732cd44-856d-4dee-8588-b29c4d4fa059\" (UID: \"1732cd44-856d-4dee-8588-b29c4d4fa059\") "
Jul 02 13:48:41.036398 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.035989    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-model-cache" (OuterVolumeSpecName: "model-cache") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.036398 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.036242    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.036398 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.036263    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.036771 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.036747    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-home" (OuterVolumeSpecName: "home") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.037357 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.037252    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-home" (OuterVolumeSpecName: "home") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.041184 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.041150    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kube-api-access-4kdr7" (OuterVolumeSpecName: "kube-api-access-4kdr7") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "kube-api-access-4kdr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:48:41.041391 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.041363    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1732cd44-856d-4dee-8588-b29c4d4fa059-kube-api-access-94vrx" (OuterVolumeSpecName: "kube-api-access-94vrx") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "kube-api-access-94vrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:48:41.041636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.041612    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1732cd44-856d-4dee-8588-b29c4d4fa059-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:48:41.042247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.042203    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:48:41.042345 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.042254    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-dshm" (OuterVolumeSpecName: "dshm") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.043743 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.043716    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-dshm" (OuterVolumeSpecName: "dshm") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.050720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.050685    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.054071 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.054051    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.102254 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.102223    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1732cd44-856d-4dee-8588-b29c4d4fa059" (UID: "1732cd44-856d-4dee-8588-b29c4d4fa059"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.107637 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.107615    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c36d3511-80e7-40e5-9839-d61bd5d08fdf" (UID: "c36d3511-80e7-40e5-9839-d61bd5d08fdf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:41.137638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137599    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137630    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137639    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdr7\" (UniqueName: \"kubernetes.io/projected/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kube-api-access-4kdr7\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137638 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137647    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vrx\" (UniqueName: \"kubernetes.io/projected/1732cd44-856d-4dee-8588-b29c4d4fa059-kube-api-access-94vrx\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137656    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137665    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137673    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137680    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36d3511-80e7-40e5-9839-d61bd5d08fdf-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137688    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cd44-856d-4dee-8588-b29c4d4fa059-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137695    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137702    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36d3511-80e7-40e5-9839-d61bd5d08fdf-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.137900 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.137709    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cd44-856d-4dee-8588-b29c4d4fa059-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:48:41.848898 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.848861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t" event={"ID":"c36d3511-80e7-40e5-9839-d61bd5d08fdf","Type":"ContainerDied","Data":"d744297e2f16f23fa07dcfa22d83bed2af5ff0093cc43c1bdcafcd9c8ccbf390"}
Jul 02 13:48:41.849057 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.848880    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"
Jul 02 13:48:41.849057 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.848959    2564 scope.go:117] "RemoveContainer" containerID="ccb74d9837d87432151e13021e0aee920d667f8fceb93265644422b93db12ce7"
Jul 02 13:48:41.851370 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.851119    2564 generic.go:356] "Generic (PLEG): container finished" podID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerID="c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495" exitCode=0
Jul 02 13:48:41.851370 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.851195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerDied","Data":"c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495"}
Jul 02 13:48:41.851370 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.851222    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"
Jul 02 13:48:41.851370 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.851265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b" event={"ID":"1732cd44-856d-4dee-8588-b29c4d4fa059","Type":"ContainerDied","Data":"ec3f88f5680795aabd9941468a99d13d8bdc49578a33346b1482d5031278aa9c"}
Jul 02 13:48:41.858586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.858569    2564 scope.go:117] "RemoveContainer" containerID="9ae82e3b37a66df889b124de204b654ae9c92210b5200fed7baf59676016913a"
Jul 02 13:48:41.866542 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.866511    2564 scope.go:117] "RemoveContainer" containerID="cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded"
Jul 02 13:48:41.871889 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.871870    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"]
Jul 02 13:48:41.874800 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.874782    2564 scope.go:117] "RemoveContainer" containerID="7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e"
Jul 02 13:48:41.876243 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.876221    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t"]
Jul 02 13:48:41.882317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.882293    2564 scope.go:117] "RemoveContainer" containerID="c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495"
Jul 02 13:48:41.889800 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.889777    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"]
Jul 02 13:48:41.890333 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.890317    2564 scope.go:117] "RemoveContainer" containerID="cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded"
Jul 02 13:48:41.890580 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:48:41.890562    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded\": container with ID starting with cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded not found: ID does not exist" containerID="cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded"
Jul 02 13:48:41.890632 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.890587    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded"} err="failed to get container status \"cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded\": rpc error: code = NotFound desc = could not find container \"cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded\": container with ID starting with cd562f51979a8e0637313bce7fffa05a4c18e587c12655b7842bec3704e3dded not found: ID does not exist"
Jul 02 13:48:41.890632 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.890601    2564 scope.go:117] "RemoveContainer" containerID="7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e"
Jul 02 13:48:41.890823 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:48:41.890804    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e\": container with ID starting with 7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e not found: ID does not exist" containerID="7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e"
Jul 02 13:48:41.890893 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.890833    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e"} err="failed to get container status \"7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e\": rpc error: code = NotFound desc = could not find container \"7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e\": container with ID starting with 7f753ddc829c28c1250d705fda5ed22efc907ffc3c621638ab6013222b41915e not found: ID does not exist"
Jul 02 13:48:41.890893 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.890855    2564 scope.go:117] "RemoveContainer" containerID="c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495"
Jul 02 13:48:41.891127 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:48:41.891110    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495\": container with ID starting with c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495 not found: ID does not exist" containerID="c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495"
Jul 02 13:48:41.891166 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.891132    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495"} err="failed to get container status \"c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495\": rpc error: code = NotFound desc = could not find container \"c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495\": container with ID starting with c0ff117a871b69de6d99c02ec427b69cf565532078cae012e600b4f4f1b61495 not found: ID does not exist"
Jul 02 13:48:41.900101 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:41.900083    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b"]
Jul 02 13:48:43.760077 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:43.760042    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" path="/var/lib/kubelet/pods/1732cd44-856d-4dee-8588-b29c4d4fa059/volumes"
Jul 02 13:48:43.760499 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:43.760485    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" path="/var/lib/kubelet/pods/c36d3511-80e7-40e5-9839-d61bd5d08fdf/volumes"
Jul 02 13:48:45.923459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:45.923432    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:48:45.932986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:45.932967    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:48:54.848920 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.848885    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"]
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849206    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="llm-d-routing-sidecar"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849216    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="llm-d-routing-sidecar"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849233    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849239    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849248    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="storage-initializer"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849254    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="storage-initializer"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849265    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849270    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849276    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="storage-initializer"
Jul 02 13:48:54.849305 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849281    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="storage-initializer"
Jul 02 13:48:54.849664 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849333    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="main"
Jul 02 13:48:54.849664 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849341    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1732cd44-856d-4dee-8588-b29c4d4fa059" containerName="llm-d-routing-sidecar"
Jul 02 13:48:54.849664 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.849348    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c36d3511-80e7-40e5-9839-d61bd5d08fdf" containerName="main"
Jul 02 13:48:54.852542 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.852513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.855157 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.855135    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:48:54.855270 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.855248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 02 13:48:54.855362 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.855344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-j8rqg\""
Jul 02 13:48:54.860045 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.859880    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"]
Jul 02 13:48:54.864426 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.864401    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.866380 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.866350    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"]
Jul 02 13:48:54.875280 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.875255    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"]
Jul 02 13:48:54.941720 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rwf\" (UniqueName: \"kubernetes.io/projected/d50df517-ff31-4170-a650-17d00b3c3817-kube-api-access-54rwf\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.941864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.941864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.941864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-model-cache\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.941864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-home\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50df517-ff31-4170-a650-17d00b3c3817-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/640ae8fa-a849-4a23-ba82-41403c010bc0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.941980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.942006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.942047 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.942025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.942316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.942074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbw2f\" (UniqueName: \"kubernetes.io/projected/640ae8fa-a849-4a23-ba82-41403c010bc0-kube-api-access-lbw2f\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:54.942316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:54.942126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-dshm\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.042706 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbw2f\" (UniqueName: \"kubernetes.io/projected/640ae8fa-a849-4a23-ba82-41403c010bc0-kube-api-access-lbw2f\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.042854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-dshm\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.042854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rwf\" (UniqueName: \"kubernetes.io/projected/d50df517-ff31-4170-a650-17d00b3c3817-kube-api-access-54rwf\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.042854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.042986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.042986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-model-cache\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.042986 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.042953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043129 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-home\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043129 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50df517-ff31-4170-a650-17d00b3c3817-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/640ae8fa-a849-4a23-ba82-41403c010bc0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043186    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043451 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043451 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043451 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043283    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043651 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043466    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-model-cache\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043651 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043480    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043651 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043496    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043803 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043759    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-home\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.043803 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043764    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.043914 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.043805    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.045191 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.045167    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-dshm\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.045664 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.045643    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.045791 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.045649    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50df517-ff31-4170-a650-17d00b3c3817-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.045851 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.045830    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/640ae8fa-a849-4a23-ba82-41403c010bc0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.050557 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.050513    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rwf\" (UniqueName: \"kubernetes.io/projected/d50df517-ff31-4170-a650-17d00b3c3817-kube-api-access-54rwf\") pod \"custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.050915 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.050897    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbw2f\" (UniqueName: \"kubernetes.io/projected/640ae8fa-a849-4a23-ba82-41403c010bc0-kube-api-access-lbw2f\") pod \"custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.162829 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.162806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.175269 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.175246    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:48:55.317095 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.317065    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"]
Jul 02 13:48:55.318645 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:48:55.318615    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50df517_ff31_4170_a650_17d00b3c3817.slice/crio-c8640d2f9b9be092a6aef6c9cb169498e0c3102581357edd947da286fc3e4ded WatchSource:0}: Error finding container c8640d2f9b9be092a6aef6c9cb169498e0c3102581357edd947da286fc3e4ded: Status 404 returned error can't find the container with id c8640d2f9b9be092a6aef6c9cb169498e0c3102581357edd947da286fc3e4ded
Jul 02 13:48:55.337626 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.337602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"]
Jul 02 13:48:55.338689 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:48:55.338659    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640ae8fa_a849_4a23_ba82_41403c010bc0.slice/crio-54a37ae9526f7d94449bc2ca6d4dd3e8f0c8d83a858f96cc2de93823ca365db1 WatchSource:0}: Error finding container 54a37ae9526f7d94449bc2ca6d4dd3e8f0c8d83a858f96cc2de93823ca365db1: Status 404 returned error can't find the container with id 54a37ae9526f7d94449bc2ca6d4dd3e8f0c8d83a858f96cc2de93823ca365db1
Jul 02 13:48:55.924603 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.924571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerStarted","Data":"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"}
Jul 02 13:48:55.925055 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.924689    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:48:55.925055 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.924718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerStarted","Data":"c8640d2f9b9be092a6aef6c9cb169498e0c3102581357edd947da286fc3e4ded"}
Jul 02 13:48:55.926174 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.926148    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" event={"ID":"640ae8fa-a849-4a23-ba82-41403c010bc0","Type":"ContainerStarted","Data":"253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90"}
Jul 02 13:48:55.926266 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:55.926192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" event={"ID":"640ae8fa-a849-4a23-ba82-41403c010bc0","Type":"ContainerStarted","Data":"54a37ae9526f7d94449bc2ca6d4dd3e8f0c8d83a858f96cc2de93823ca365db1"}
Jul 02 13:48:56.938310 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:56.938263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerStarted","Data":"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"}
Jul 02 13:48:58.950392 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:58.950356    2564 generic.go:356] "Generic (PLEG): container finished" podID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerID="253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90" exitCode=0
Jul 02 13:48:58.950833 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:58.950433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" event={"ID":"640ae8fa-a849-4a23-ba82-41403c010bc0","Type":"ContainerDied","Data":"253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90"}
Jul 02 13:48:59.958545 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:59.958492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" event={"ID":"640ae8fa-a849-4a23-ba82-41403c010bc0","Type":"ContainerStarted","Data":"59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400"}
Jul 02 13:48:59.982781 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:48:59.982729    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podStartSLOduration=2.500247987 podStartE2EDuration="5.982713631s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.482465644s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:48:59.981322053 +0000 UTC m=+2114.712360211" watchObservedRunningTime="2026-07-02 13:48:59.982713631 +0000 UTC m=+2114.713751760"
Jul 02 13:49:00.966097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:00.966063    2564 generic.go:356] "Generic (PLEG): container finished" podID="d50df517-ff31-4170-a650-17d00b3c3817" containerID="eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac" exitCode=0
Jul 02 13:49:00.966478 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:00.966119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerDied","Data":"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"}
Jul 02 13:49:01.974105 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:01.974070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerStarted","Data":"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"}
Jul 02 13:49:02.001340 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:02.001294    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podStartSLOduration=3.563291527 podStartE2EDuration="8.001279258s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.437987731s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:02.00010581 +0000 UTC m=+2116.731143969" watchObservedRunningTime="2026-07-02 13:49:02.001279258 +0000 UTC m=+2116.732317418"
Jul 02 13:49:05.163461 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.163422    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:49:05.163912 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.163498    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:49:05.164890 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.164854    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:49:05.176132 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.176099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:49:05.176293 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.176163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:49:05.176919 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.176897    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:49:05.177494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:05.177468    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:49:15.163268 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:15.163197    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:49:15.176317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:15.176286    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:49:25.164128 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:25.164067    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:49:25.176430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:25.176393    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:49:35.163407 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:35.163365    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:49:35.175738 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:35.175694    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:49:37.489087 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:49:37.489030    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:49:37.491620 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:49:37.489819    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:49:38.176038 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:38.175952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:49:38.176218 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:38.175964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:49:45.163604 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:45.163552    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:49:45.175824 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:45.175785    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:49:55.163418 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:55.163360    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:49:55.175923 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:49:55.175892    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:50:05.164197 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:05.164149    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:50:05.175792 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:05.175751    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:50:15.164167 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:15.164116    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:50:15.176308 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:15.176278    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:50:25.164106 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:25.164048    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:50:25.176249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:25.176205    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:50:35.163413 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:35.163365    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:50:35.178166 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:35.178117    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:50:45.163820 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:45.163771    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:50:45.175543 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:45.175496    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:50:55.163979 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:55.163915    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:50:55.175993 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:50:55.175962    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:51:05.163557 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:05.163498    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:51:05.175848 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:05.175807    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:51:15.163945 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:15.163849    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:51:15.176395 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:15.176364    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 02 13:51:25.163208 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:25.163159    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 02 13:51:25.185708 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:25.185681    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:51:25.193473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:25.193446    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:51:35.173709 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:35.173682    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:51:35.185407 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:35.185386    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:51:41.177466 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:51:41.177418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:51:41.177466 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:51:41.177418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:51:41.848347 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:41.848320    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:51:41.848589 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:51:41.848322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:52:22.556767 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:22.556727    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"]
Jul 02 13:52:22.559373 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:22.559347    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"]
Jul 02 13:52:37.582180 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:37.582120    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main" containerID="cri-o://cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9" gracePeriod=45
Jul 02 13:52:37.584746 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:37.584710    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main" containerID="cri-o://59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400" gracePeriod=45
Jul 02 13:52:38.332727 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.332673    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="llm-d-routing-sidecar" containerID="cri-o://274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272" gracePeriod=45
Jul 02 13:52:38.466013 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.465987    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:52:38.575833 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.575810    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:52:38.590804 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.590779    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-model-cache\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.590830    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/640ae8fa-a849-4a23-ba82-41403c010bc0-tls-certs\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.590865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-tmp-dir\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.590924    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbw2f\" (UniqueName: \"kubernetes.io/projected/640ae8fa-a849-4a23-ba82-41403c010bc0-kube-api-access-lbw2f\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.590955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-dshm\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.591034    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-kserve-provision-location\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.591043    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.591160 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.591072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-home\") pod \"640ae8fa-a849-4a23-ba82-41403c010bc0\" (UID: \"640ae8fa-a849-4a23-ba82-41403c010bc0\") "
Jul 02 13:52:38.591586 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.591380    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.591850 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.591816    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-home" (OuterVolumeSpecName: "home") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.592980 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.592954    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/640ae8fa-a849-4a23-ba82-41403c010bc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:52:38.593094 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.593048    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-dshm" (OuterVolumeSpecName: "dshm") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.593359 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.593338    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640ae8fa-a849-4a23-ba82-41403c010bc0-kube-api-access-lbw2f" (OuterVolumeSpecName: "kube-api-access-lbw2f") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "kube-api-access-lbw2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:52:38.606468 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.606440    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.665614 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.665585    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "640ae8fa-a849-4a23-ba82-41403c010bc0" (UID: "640ae8fa-a849-4a23-ba82-41403c010bc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.691723 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691701    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54rwf\" (UniqueName: \"kubernetes.io/projected/d50df517-ff31-4170-a650-17d00b3c3817-kube-api-access-54rwf\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.691812 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691736    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-model-cache\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.691812 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-home\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.691927 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691813    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50df517-ff31-4170-a650-17d00b3c3817-tls-certs\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.691927 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-tmp-dir\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.691927 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-kserve-provision-location\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.691927 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-dshm\") pod \"d50df517-ff31-4170-a650-17d00b3c3817\" (UID: \"d50df517-ff31-4170-a650-17d00b3c3817\") "
Jul 02 13:52:38.692147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.691999    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-model-cache" (OuterVolumeSpecName: "model-cache") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.692234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692206    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692240    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692255    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/640ae8fa-a849-4a23-ba82-41403c010bc0-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692269    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692282    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbw2f\" (UniqueName: \"kubernetes.io/projected/640ae8fa-a849-4a23-ba82-41403c010bc0-kube-api-access-lbw2f\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692293    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/640ae8fa-a849-4a23-ba82-41403c010bc0-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692304    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.692764 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.692480    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-home" (OuterVolumeSpecName: "home") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.693919 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.693894    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50df517-ff31-4170-a650-17d00b3c3817-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:52:38.694237 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.694210    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-dshm" (OuterVolumeSpecName: "dshm") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.694307 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.694219    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50df517-ff31-4170-a650-17d00b3c3817-kube-api-access-54rwf" (OuterVolumeSpecName: "kube-api-access-54rwf") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "kube-api-access-54rwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:52:38.705501 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.705480    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.752733 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.752706    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d50df517-ff31-4170-a650-17d00b3c3817" (UID: "d50df517-ff31-4170-a650-17d00b3c3817"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:38.793176 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.793145    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rwf\" (UniqueName: \"kubernetes.io/projected/d50df517-ff31-4170-a650-17d00b3c3817-kube-api-access-54rwf\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.793176 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.793177    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.793286 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.793187    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50df517-ff31-4170-a650-17d00b3c3817-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.793286 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.793195    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.793286 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.793205    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:38.793286 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:38.793212    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50df517-ff31-4170-a650-17d00b3c3817-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:52:39.141406 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141377    2564 generic.go:356] "Generic (PLEG): container finished" podID="d50df517-ff31-4170-a650-17d00b3c3817" containerID="cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9" exitCode=0
Jul 02 13:52:39.141406 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141401    2564 generic.go:356] "Generic (PLEG): container finished" podID="d50df517-ff31-4170-a650-17d00b3c3817" containerID="274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272" exitCode=0
Jul 02 13:52:39.141644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141439    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"
Jul 02 13:52:39.141644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141462    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerDied","Data":"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"}
Jul 02 13:52:39.141644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerDied","Data":"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"}
Jul 02 13:52:39.141644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p" event={"ID":"d50df517-ff31-4170-a650-17d00b3c3817","Type":"ContainerDied","Data":"c8640d2f9b9be092a6aef6c9cb169498e0c3102581357edd947da286fc3e4ded"}
Jul 02 13:52:39.141644 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.141565    2564 scope.go:117] "RemoveContainer" containerID="cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"
Jul 02 13:52:39.143269 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.143246    2564 generic.go:356] "Generic (PLEG): container finished" podID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerID="59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400" exitCode=0
Jul 02 13:52:39.143357 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.143279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" event={"ID":"640ae8fa-a849-4a23-ba82-41403c010bc0","Type":"ContainerDied","Data":"59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400"}
Jul 02 13:52:39.143357 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.143300    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"
Jul 02 13:52:39.143357 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.143350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d" event={"ID":"640ae8fa-a849-4a23-ba82-41403c010bc0","Type":"ContainerDied","Data":"54a37ae9526f7d94449bc2ca6d4dd3e8f0c8d83a858f96cc2de93823ca365db1"}
Jul 02 13:52:39.150619 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.150583    2564 scope.go:117] "RemoveContainer" containerID="eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"
Jul 02 13:52:39.158616 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.158595    2564 scope.go:117] "RemoveContainer" containerID="274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"
Jul 02 13:52:39.166348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.166332    2564 scope.go:117] "RemoveContainer" containerID="cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"
Jul 02 13:52:39.166746 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:52:39.166721    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9\": container with ID starting with cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9 not found: ID does not exist" containerID="cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"
Jul 02 13:52:39.166835 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.166756    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"} err="failed to get container status \"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9\": rpc error: code = NotFound desc = could not find container \"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9\": container with ID starting with cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9 not found: ID does not exist"
Jul 02 13:52:39.166835 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.166778    2564 scope.go:117] "RemoveContainer" containerID="eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"
Jul 02 13:52:39.167158 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:52:39.167124    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac\": container with ID starting with eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac not found: ID does not exist" containerID="eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"
Jul 02 13:52:39.167244 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.167168    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"} err="failed to get container status \"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac\": rpc error: code = NotFound desc = could not find container \"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac\": container with ID starting with eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac not found: ID does not exist"
Jul 02 13:52:39.167244 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.167194    2564 scope.go:117] "RemoveContainer" containerID="274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"
Jul 02 13:52:39.167474 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:52:39.167456    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272\": container with ID starting with 274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272 not found: ID does not exist" containerID="274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"
Jul 02 13:52:39.167521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.167478    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"} err="failed to get container status \"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272\": rpc error: code = NotFound desc = could not find container \"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272\": container with ID starting with 274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272 not found: ID does not exist"
Jul 02 13:52:39.167521 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.167493    2564 scope.go:117] "RemoveContainer" containerID="cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"
Jul 02 13:52:39.167800 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.167782    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9"} err="failed to get container status \"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9\": rpc error: code = NotFound desc = could not find container \"cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9\": container with ID starting with cf91a6b77a4416823c0caf615e3fe7c1c163dc1754436fcd6bbcdafe49d3ddf9 not found: ID does not exist"
Jul 02 13:52:39.167883 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.167802    2564 scope.go:117] "RemoveContainer" containerID="eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"
Jul 02 13:52:39.168078 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.168061    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac"} err="failed to get container status \"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac\": rpc error: code = NotFound desc = could not find container \"eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac\": container with ID starting with eceb4aa7770bbd20fec169af85fde80e3f80f96d74c0e6da13e7b03170f4f2ac not found: ID does not exist"
Jul 02 13:52:39.168149 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.168080    2564 scope.go:117] "RemoveContainer" containerID="274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"
Jul 02 13:52:39.168317 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.168294    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272"} err="failed to get container status \"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272\": rpc error: code = NotFound desc = could not find container \"274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272\": container with ID starting with 274e76451ebaa58c9a572b87c179ccd8f9fa98cd18e8ad7c29be413b4cfb3272 not found: ID does not exist"
Jul 02 13:52:39.168368 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.168319    2564 scope.go:117] "RemoveContainer" containerID="59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400"
Jul 02 13:52:39.168409 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.168391    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"]
Jul 02 13:52:39.171868 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.171847    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p"]
Jul 02 13:52:39.176072 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.176057    2564 scope.go:117] "RemoveContainer" containerID="253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90"
Jul 02 13:52:39.184152 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.184131    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"]
Jul 02 13:52:39.185037 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.185019    2564 scope.go:117] "RemoveContainer" containerID="59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400"
Jul 02 13:52:39.185715 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:52:39.185390    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400\": container with ID starting with 59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400 not found: ID does not exist" containerID="59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400"
Jul 02 13:52:39.185715 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.185420    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400"} err="failed to get container status \"59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400\": rpc error: code = NotFound desc = could not find container \"59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400\": container with ID starting with 59cfe72348901815d369cc01a1adf489cbb8e8d9a70f699d9d1dd1a2a0702400 not found: ID does not exist"
Jul 02 13:52:39.185715 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.185441    2564 scope.go:117] "RemoveContainer" containerID="253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90"
Jul 02 13:52:39.185914 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:52:39.185801    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90\": container with ID starting with 253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90 not found: ID does not exist" containerID="253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90"
Jul 02 13:52:39.185914 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.185825    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90"} err="failed to get container status \"253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90\": rpc error: code = NotFound desc = could not find container \"253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90\": container with ID starting with 253c276057d2972386e38051b08c5a04513762c2e2fcb8c96dcffcd01730da90 not found: ID does not exist"
Jul 02 13:52:39.187821 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.187803    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d"]
Jul 02 13:52:39.764747 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.761491    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" path="/var/lib/kubelet/pods/640ae8fa-a849-4a23-ba82-41403c010bc0/volumes"
Jul 02 13:52:39.764747 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:52:39.762138    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50df517-ff31-4170-a650-17d00b3c3817" path="/var/lib/kubelet/pods/d50df517-ff31-4170-a650-17d00b3c3817/volumes"
Jul 02 13:53:03.230541 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230492    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"]
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230835    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="llm-d-routing-sidecar"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230847    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="llm-d-routing-sidecar"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230856    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="storage-initializer"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230861    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="storage-initializer"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230870    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230876    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230889    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="storage-initializer"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230894    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="storage-initializer"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230903    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main"
Jul 02 13:53:03.230940 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230908    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main"
Jul 02 13:53:03.231255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230961    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="llm-d-routing-sidecar"
Jul 02 13:53:03.231255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230970    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="640ae8fa-a849-4a23-ba82-41403c010bc0" containerName="main"
Jul 02 13:53:03.231255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.230976    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50df517-ff31-4170-a650-17d00b3c3817" containerName="main"
Jul 02 13:53:03.234323 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.234301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.237145 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.237114    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:53:03.237426 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.237404    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-ksdvd\""
Jul 02 13:53:03.237548 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.237442    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 02 13:53:03.247143 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.247115    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"]
Jul 02 13:53:03.263450 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.263425    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"]
Jul 02 13:53:03.267234 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.267215    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288343 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-dshm\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.288459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-home\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288459 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fts5q\" (UniqueName: \"kubernetes.io/projected/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kube-api-access-fts5q\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db956b-1633-4341-8139-6c1bd09f73bd-tls-certs\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.288656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-home\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.288656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-model-cache\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.288656 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6q8s\" (UniqueName: \"kubernetes.io/projected/f3db956b-1633-4341-8139-6c1bd09f73bd-kube-api-access-c6q8s\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.288842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.288842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.288842 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.288759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.299203 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.299185    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"]
Jul 02 13:53:03.389982 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.389954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fts5q\" (UniqueName: \"kubernetes.io/projected/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kube-api-access-fts5q\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.389988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db956b-1633-4341-8139-6c1bd09f73bd-tls-certs\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-home\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-model-cache\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6q8s\" (UniqueName: \"kubernetes.io/projected/f3db956b-1633-4341-8139-6c1bd09f73bd-kube-api-access-c6q8s\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-dshm\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-home\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390279    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390430 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390444    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-home\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390472    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-model-cache\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390564    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390591    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390617    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390774    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-home\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.390837 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390801    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.391202 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.390915    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.392460 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.392435    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.392674 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.392657    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-dshm\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.392751 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.392704    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db956b-1633-4341-8139-6c1bd09f73bd-tls-certs\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.393217 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.393197    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.419248 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.419197    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fts5q\" (UniqueName: \"kubernetes.io/projected/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kube-api-access-fts5q\") pod \"router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.420057 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.420029    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6q8s\" (UniqueName: \"kubernetes.io/projected/f3db956b-1633-4341-8139-6c1bd09f73bd-kube-api-access-c6q8s\") pod \"router-with-refs-pd-test-kserve-554644558c-85lw4\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.547504 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.547443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:03.576911 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.576884    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:03.695933 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.695905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"]
Jul 02 13:53:03.697688 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:53:03.697657    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3db956b_1633_4341_8139_6c1bd09f73bd.slice/crio-c5891e79ca9d29f9010272d3362822056f53888e6f97c55beb28aef4a1871ac9 WatchSource:0}: Error finding container c5891e79ca9d29f9010272d3362822056f53888e6f97c55beb28aef4a1871ac9: Status 404 returned error can't find the container with id c5891e79ca9d29f9010272d3362822056f53888e6f97c55beb28aef4a1871ac9
Jul 02 13:53:03.725541 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:03.725508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"]
Jul 02 13:53:03.732790 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:53:03.732761    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cd2d4c_8b41_45e3_a653_5a238a7d5e92.slice/crio-ac394633d4a386e55461d375a3d53ad19e9cede71b508135726fd36a893487d2 WatchSource:0}: Error finding container ac394633d4a386e55461d375a3d53ad19e9cede71b508135726fd36a893487d2: Status 404 returned error can't find the container with id ac394633d4a386e55461d375a3d53ad19e9cede71b508135726fd36a893487d2
Jul 02 13:53:04.274061 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:04.274020    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" event={"ID":"19cd2d4c-8b41-45e3-a653-5a238a7d5e92","Type":"ContainerStarted","Data":"a9b44674694c6781cb724df6875832b2cc18176e5c72e401dbef2aeff6199ff5"}
Jul 02 13:53:04.274508 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:04.274082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" event={"ID":"19cd2d4c-8b41-45e3-a653-5a238a7d5e92","Type":"ContainerStarted","Data":"ac394633d4a386e55461d375a3d53ad19e9cede71b508135726fd36a893487d2"}
Jul 02 13:53:04.275654 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:04.275624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerStarted","Data":"c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2"}
Jul 02 13:53:04.275780 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:04.275668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerStarted","Data":"c5891e79ca9d29f9010272d3362822056f53888e6f97c55beb28aef4a1871ac9"}
Jul 02 13:53:04.275780 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:04.275691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:05.291342 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:05.291258    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerStarted","Data":"0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536"}
Jul 02 13:53:08.310476 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:08.310445    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerID="0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536" exitCode=0
Jul 02 13:53:08.310954 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:08.310544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerDied","Data":"0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536"}
Jul 02 13:53:08.312127 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:08.312104    2564 generic.go:356] "Generic (PLEG): container finished" podID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerID="a9b44674694c6781cb724df6875832b2cc18176e5c72e401dbef2aeff6199ff5" exitCode=0
Jul 02 13:53:08.312245 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:08.312162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" event={"ID":"19cd2d4c-8b41-45e3-a653-5a238a7d5e92","Type":"ContainerDied","Data":"a9b44674694c6781cb724df6875832b2cc18176e5c72e401dbef2aeff6199ff5"}
Jul 02 13:53:09.320865 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:09.320824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" event={"ID":"19cd2d4c-8b41-45e3-a653-5a238a7d5e92","Type":"ContainerStarted","Data":"9d90acb8d9c138f563c68217b4e369ed6adc54325b70e5cfcf01ad12a57c45e7"}
Jul 02 13:53:09.323246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:09.323211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerStarted","Data":"57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c"}
Jul 02 13:53:09.344614 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:09.344496    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podStartSLOduration=2.321608657 podStartE2EDuration="6.344475101s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.022866444s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:53:09.342851211 +0000 UTC m=+2364.073889377" watchObservedRunningTime="2026-07-02 13:53:09.344475101 +0000 UTC m=+2364.075513232"
Jul 02 13:53:09.365690 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:09.365644    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podStartSLOduration=2.810886321 podStartE2EDuration="6.365619917s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554733596s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:53:09.3648769 +0000 UTC m=+2364.095915058" watchObservedRunningTime="2026-07-02 13:53:09.365619917 +0000 UTC m=+2364.096658050"
Jul 02 13:53:13.548461 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.548414    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:13.548917 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.548542    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:13.550222 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.550182    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:53:13.562069 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.562048    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:53:13.577615 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.577594    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:13.577712 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.577645    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:53:13.579172 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:13.579143    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:53:23.548392 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:23.548342    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:53:23.577954 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:23.577909    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:53:33.548598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:33.548519    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:53:33.577851 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:33.577817    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:53:43.548451 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:43.548373    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:53:43.577393 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:43.577354    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:53:44.849469 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:53:44.849401    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:53:44.850278 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:53:44.850251    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:53:45.555025 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:45.554990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:53:45.555225 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:45.554996    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:53:45.961646 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:45.961617    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:53:45.970473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:45.970450    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:53:53.548351 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:53.548298    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:53:53.577813 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:53:53.577776    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:54:03.548513 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:03.548465    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:54:03.577367 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:03.577328    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:54:13.548678 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:13.548572    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:54:13.577429 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:13.577391    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:54:23.548473 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:23.548424    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:54:23.577518 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:23.577484    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:54:33.548432 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:33.548385    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:54:33.577748 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:33.577719    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:54:43.548610 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:43.548555    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:54:43.577840 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:43.577811    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:54:53.548765 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:53.548718    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:54:53.577316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:54:53.577274    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:55:03.547879 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:03.547828    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:55:03.577771 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:03.577738    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:55:13.548573 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:13.548509    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:55:13.577215 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:13.577176    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:55:23.547890 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:23.547844    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:55:23.578006 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:23.577969    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:55:33.548146 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:33.548100    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 02 13:55:33.577344 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:33.577293    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 02 13:55:43.557378 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:43.557295    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:55:43.570212 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:43.570191    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:55:43.586739 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:43.586713    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:55:43.595073 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:43.595051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:55:48.556054 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:55:48.556009    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:55:48.556054 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:55:48.556009    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:55:49.238872 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:49.238842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:55:49.239050 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:55:49.238842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:56:03.514145 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:03.514104    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"]
Jul 02 13:56:03.519750 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:03.519721    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"]
Jul 02 13:56:18.541205 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:18.541158    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main" containerID="cri-o://9d90acb8d9c138f563c68217b4e369ed6adc54325b70e5cfcf01ad12a57c45e7" gracePeriod=45
Jul 02 13:56:18.544903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:18.544868    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main" containerID="cri-o://57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c" gracePeriod=45
Jul 02 13:56:19.291494 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.291461    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="llm-d-routing-sidecar" containerID="cri-o://c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2" gracePeriod=45
Jul 02 13:56:19.404596 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.404566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" event={"ID":"19cd2d4c-8b41-45e3-a653-5a238a7d5e92","Type":"ContainerDied","Data":"9d90acb8d9c138f563c68217b4e369ed6adc54325b70e5cfcf01ad12a57c45e7"}
Jul 02 13:56:19.404719 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.404567    2564 generic.go:356] "Generic (PLEG): container finished" podID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerID="9d90acb8d9c138f563c68217b4e369ed6adc54325b70e5cfcf01ad12a57c45e7" exitCode=0
Jul 02 13:56:19.406994 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.406970    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerID="57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c" exitCode=0
Jul 02 13:56:19.407111 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.407045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerDied","Data":"57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c"}
Jul 02 13:56:19.423296 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.423276    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:56:19.536177 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.536146    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:56:19.578271 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-dshm\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.578725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578284    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tmp-dir\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.578725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578325    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fts5q\" (UniqueName: \"kubernetes.io/projected/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kube-api-access-fts5q\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.578725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-home\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.578725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578427    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-model-cache\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.578725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578452    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tls-certs\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.578725 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.578481    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kserve-provision-location\") pod \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\" (UID: \"19cd2d4c-8b41-45e3-a653-5a238a7d5e92\") "
Jul 02 13:56:19.579048 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.579010    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-model-cache" (OuterVolumeSpecName: "model-cache") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.579179 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.579152    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-home" (OuterVolumeSpecName: "home") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.580651 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.580616    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-dshm" (OuterVolumeSpecName: "dshm") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.580919 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.580895    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:56:19.581002 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.580987    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kube-api-access-fts5q" (OuterVolumeSpecName: "kube-api-access-fts5q") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "kube-api-access-fts5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:56:19.598694 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.598670    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.646838 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.646808    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19cd2d4c-8b41-45e3-a653-5a238a7d5e92" (UID: "19cd2d4c-8b41-45e3-a653-5a238a7d5e92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.679159 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-kserve-provision-location\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679159 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679166    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-tmp-dir\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679318 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679208    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-home\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679318 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679233    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6q8s\" (UniqueName: \"kubernetes.io/projected/f3db956b-1633-4341-8139-6c1bd09f73bd-kube-api-access-c6q8s\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679318 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-dshm\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679318 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679276    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-model-cache\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679318 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679306    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db956b-1633-4341-8139-6c1bd09f73bd-tls-certs\") pod \"f3db956b-1633-4341-8139-6c1bd09f73bd\" (UID: \"f3db956b-1633-4341-8139-6c1bd09f73bd\") "
Jul 02 13:56:19.679640 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679617    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.679640 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679617    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.679770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679645    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fts5q\" (UniqueName: \"kubernetes.io/projected/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kube-api-access-fts5q\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.679770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679660    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.679770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679679    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.679770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679694    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.679770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679709    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.679770 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679720    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cd2d4c-8b41-45e3-a653-5a238a7d5e92-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.680097 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.679853    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-home" (OuterVolumeSpecName: "home") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.681247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.681227    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-dshm" (OuterVolumeSpecName: "dshm") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.681685 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.681664    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3db956b-1633-4341-8139-6c1bd09f73bd-kube-api-access-c6q8s" (OuterVolumeSpecName: "kube-api-access-c6q8s") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "kube-api-access-c6q8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:56:19.681948 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.681930    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3db956b-1633-4341-8139-6c1bd09f73bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:56:19.692673 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.692649    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.738592 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.738552    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3db956b-1633-4341-8139-6c1bd09f73bd" (UID: "f3db956b-1633-4341-8139-6c1bd09f73bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:19.780784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780726    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.780784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780757    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.780784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780768    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.780784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780777    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6q8s\" (UniqueName: \"kubernetes.io/projected/f3db956b-1633-4341-8139-6c1bd09f73bd-kube-api-access-c6q8s\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.780784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780784    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.780784 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780793    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db956b-1633-4341-8139-6c1bd09f73bd-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:19.781026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:19.780800    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db956b-1633-4341-8139-6c1bd09f73bd-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:56:20.413581 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.413552    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerID="c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2" exitCode=0
Jul 02 13:56:20.413740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.413601    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"
Jul 02 13:56:20.413740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.413637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerDied","Data":"c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2"}
Jul 02 13:56:20.413740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.413690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4" event={"ID":"f3db956b-1633-4341-8139-6c1bd09f73bd","Type":"ContainerDied","Data":"c5891e79ca9d29f9010272d3362822056f53888e6f97c55beb28aef4a1871ac9"}
Jul 02 13:56:20.413740 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.413714    2564 scope.go:117] "RemoveContainer" containerID="57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c"
Jul 02 13:56:20.415385 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.415362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv" event={"ID":"19cd2d4c-8b41-45e3-a653-5a238a7d5e92","Type":"ContainerDied","Data":"ac394633d4a386e55461d375a3d53ad19e9cede71b508135726fd36a893487d2"}
Jul 02 13:56:20.415490 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.415404    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"
Jul 02 13:56:20.422300 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.422285    2564 scope.go:117] "RemoveContainer" containerID="0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536"
Jul 02 13:56:20.430205 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.430188    2564 scope.go:117] "RemoveContainer" containerID="c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2"
Jul 02 13:56:20.437278 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.437023    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"]
Jul 02 13:56:20.439144 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439122    2564 scope.go:117] "RemoveContainer" containerID="57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c"
Jul 02 13:56:20.439418 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:56:20.439397    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c\": container with ID starting with 57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c not found: ID does not exist" containerID="57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c"
Jul 02 13:56:20.439498 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439424    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c"} err="failed to get container status \"57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c\": rpc error: code = NotFound desc = could not find container \"57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c\": container with ID starting with 57be6ab54252e7be07ab0291d1eacea4ca19719693b56f5b162ce6894491e60c not found: ID does not exist"
Jul 02 13:56:20.439498 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439439    2564 scope.go:117] "RemoveContainer" containerID="0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536"
Jul 02 13:56:20.439688 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:56:20.439671    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536\": container with ID starting with 0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536 not found: ID does not exist" containerID="0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536"
Jul 02 13:56:20.439754 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439692    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536"} err="failed to get container status \"0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536\": rpc error: code = NotFound desc = could not find container \"0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536\": container with ID starting with 0835519aaa85d01c430f454499a535d24a240aed74bdcc928c728daffba53536 not found: ID does not exist"
Jul 02 13:56:20.439754 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439705    2564 scope.go:117] "RemoveContainer" containerID="c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2"
Jul 02 13:56:20.439908 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:56:20.439889    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2\": container with ID starting with c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2 not found: ID does not exist" containerID="c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2"
Jul 02 13:56:20.440001 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439908    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2"} err="failed to get container status \"c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2\": rpc error: code = NotFound desc = could not find container \"c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2\": container with ID starting with c3d4045bc68b39a6fbd8ff413986a5fae2ec00d4102cd95487f0d73da94d85f2 not found: ID does not exist"
Jul 02 13:56:20.440001 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.439923    2564 scope.go:117] "RemoveContainer" containerID="9d90acb8d9c138f563c68217b4e369ed6adc54325b70e5cfcf01ad12a57c45e7"
Jul 02 13:56:20.440309 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.440288    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4"]
Jul 02 13:56:20.447346 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.447332    2564 scope.go:117] "RemoveContainer" containerID="a9b44674694c6781cb724df6875832b2cc18176e5c72e401dbef2aeff6199ff5"
Jul 02 13:56:20.452977 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.452958    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"]
Jul 02 13:56:20.458883 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:20.458862    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv"]
Jul 02 13:56:21.759797 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:21.759766    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" path="/var/lib/kubelet/pods/19cd2d4c-8b41-45e3-a653-5a238a7d5e92/volumes"
Jul 02 13:56:21.760173 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:21.760160    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" path="/var/lib/kubelet/pods/f3db956b-1633-4341-8139-6c1bd09f73bd/volumes"
Jul 02 13:56:56.307154 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307123    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"]
Jul 02 13:56:56.307635 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307615    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="llm-d-routing-sidecar"
Jul 02 13:56:56.307691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307639    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="llm-d-routing-sidecar"
Jul 02 13:56:56.307691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307658    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main"
Jul 02 13:56:56.307691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307666    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main"
Jul 02 13:56:56.307691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307681    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="storage-initializer"
Jul 02 13:56:56.307691 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307689    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="storage-initializer"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307707    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="storage-initializer"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307715    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="storage-initializer"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307725    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307733    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307805    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cd2d4c-8b41-45e3-a653-5a238a7d5e92" containerName="main"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307819    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="main"
Jul 02 13:56:56.307846 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.307828    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3db956b-1633-4341-8139-6c1bd09f73bd" containerName="llm-d-routing-sidecar"
Jul 02 13:56:56.311021 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.311004    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.313680 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.313655    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:56:56.313787 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.313712    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 02 13:56:56.321815 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.321792    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"]
Jul 02 13:56:56.366928 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.366883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.367092 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.366931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jz8z\" (UniqueName: \"kubernetes.io/projected/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kube-api-access-8jz8z\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.367092 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.366991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.367092 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.367063    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.367255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.367104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.367255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.367168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.367255 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.367201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.467578 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.467735 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.467786 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467761    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.467845 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.467903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.467903 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jz8z\" (UniqueName: \"kubernetes.io/projected/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kube-api-access-8jz8z\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.468019 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.468019 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.467938    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.468149 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.468116    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.468249 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.468220    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.468322 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.468286    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.470026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.470000    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.470121 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.470092    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.476316 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.476291    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jz8z\" (UniqueName: \"kubernetes.io/projected/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kube-api-access-8jz8z\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.622474 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.622403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:56:56.766102 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:56.766072    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"]
Jul 02 13:56:56.769337 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:56:56.769313    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d70523c_c9be_4ba9_baa8_29ec77d3bd0f.slice/crio-a5439825c4e5c55e008c0417832dee6e9411c32059ed9da4aeb3b41f7b7a5889 WatchSource:0}: Error finding container a5439825c4e5c55e008c0417832dee6e9411c32059ed9da4aeb3b41f7b7a5889: Status 404 returned error can't find the container with id a5439825c4e5c55e008c0417832dee6e9411c32059ed9da4aeb3b41f7b7a5889
Jul 02 13:56:57.606687 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:57.606653    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" event={"ID":"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f","Type":"ContainerStarted","Data":"faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7"}
Jul 02 13:56:57.607087 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:56:57.606700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" event={"ID":"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f","Type":"ContainerStarted","Data":"a5439825c4e5c55e008c0417832dee6e9411c32059ed9da4aeb3b41f7b7a5889"}
Jul 02 13:57:02.632891 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:02.632855    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerID="faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7" exitCode=0
Jul 02 13:57:02.633363 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:02.632937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" event={"ID":"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f","Type":"ContainerDied","Data":"faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7"}
Jul 02 13:57:03.640625 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:03.640596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" event={"ID":"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f","Type":"ContainerStarted","Data":"f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00"}
Jul 02 13:57:03.666714 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:03.666661    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podStartSLOduration=2.808094073 podStartE2EDuration="7.666646352s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.858552279s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:57:03.66265386 +0000 UTC m=+2598.393692015" watchObservedRunningTime="2026-07-02 13:57:03.666646352 +0000 UTC m=+2598.397684487"
Jul 02 13:57:04.623353 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:04.623319    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:57:06.623446 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:06.623412    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:57:06.625062 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:06.625038    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:57:16.622830 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:16.622741    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:57:26.622864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:26.622828    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:57:36.623747 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:36.623709    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:57:46.623257 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:46.623211    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:57:52.239552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:57:52.239487    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:57:52.239552 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:57:52.239487    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:57:52.911864 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:52.911839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:57:52.912056 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:52.911839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:57:56.622884 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:57:56.622844    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:58:06.623613 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:06.623572    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:58:16.623125 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:16.623081    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:58:26.623137 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:26.623088    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:58:29.373335 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.373293    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 02 13:58:29.376985 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.376961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.381001 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.380980    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 02 13:58:29.382320 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.382302    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b\""
Jul 02 13:58:29.389583 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.389562    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 02 13:58:29.434171 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.434140    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 02 13:58:29.437441 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.437421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.453072 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.453050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 02 13:58:29.507247 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.507209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77grz\" (UniqueName: \"kubernetes.io/projected/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-kube-api-access-77grz\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.507501 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.507472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.507670 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.507516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.507739 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.507721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.507800 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.507764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.507800 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.507819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608348 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608478 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.608478 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.608478 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77grz\" (UniqueName: \"kubernetes.io/projected/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-kube-api-access-77grz\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608620 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608620 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608717 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7kq\" (UniqueName: \"kubernetes.io/projected/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-kube-api-access-fc7kq\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.608717 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.608717 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608695    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608880 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608756    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608880 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.608880 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.608880 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.609071 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.608898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.609201 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.609181    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.610768 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.610748    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.611007 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.610990    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.617404 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.617383    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77grz\" (UniqueName: \"kubernetes.io/projected/d3f69700-0389-4c20-8fd5-d1d9d1608bf9-kube-api-access-77grz\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"d3f69700-0389-4c20-8fd5-d1d9d1608bf9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 13:58:29.709590 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.709745 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.709745 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.709745 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7kq\" (UniqueName: \"kubernetes.io/projected/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-kube-api-access-fc7kq\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.709919 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.709919 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.710026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709930    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.710026 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.709941    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.710246 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.710217    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.711881 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.711853    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.712116 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.712100    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:29.718854 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:29.718835    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7kq\" (UniqueName: \"kubernetes.io/projected/5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7-kube-api-access-fc7kq\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 13:58:36.623402 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:36.623348    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:58:45.996103 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:45.996041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:58:46.007678 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:46.007658    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 13:58:46.623617 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:46.623567    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:58:56.634037 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:56.634008    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:58:56.642738 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:58:56.642710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:59:19.387463 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:19.387429    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"]
Jul 02 13:59:34.411675 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:34.411613    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" containerID="cri-o://f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00" gracePeriod=45
Jul 02 13:59:34.634598 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:34.634555    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 02 13:59:35.064443 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.064419    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:59:35.200985 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.200947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-model-cache\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201153 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201002    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-home\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201153 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201029    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tls-certs\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201153 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201055    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jz8z\" (UniqueName: \"kubernetes.io/projected/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kube-api-access-8jz8z\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201153 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201074    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tmp-dir\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201153 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-dshm\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201153 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201130    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kserve-provision-location\") pod \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\" (UID: \"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f\") "
Jul 02 13:59:35.201479 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201242    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:35.201578 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201487    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.201766 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.201739    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-home" (OuterVolumeSpecName: "home") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:35.203223 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.203189    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:59:35.203339 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.203221    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-dshm" (OuterVolumeSpecName: "dshm") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:35.203339 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.203314    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kube-api-access-8jz8z" (OuterVolumeSpecName: "kube-api-access-8jz8z") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "kube-api-access-8jz8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:59:35.220814 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.220792    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:35.266636 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.266609    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" (UID: "8d70523c-c9be-4ba9-baa8-29ec77d3bd0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:35.302104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.302068    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.302104 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.302103    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.302239 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.302119    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jz8z\" (UniqueName: \"kubernetes.io/projected/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kube-api-access-8jz8z\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.302239 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.302130    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.302239 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.302142    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.302239 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.302154    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 13:59:35.451886 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.451809    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerID="f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00" exitCode=0
Jul 02 13:59:35.451886 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.451847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" event={"ID":"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f","Type":"ContainerDied","Data":"f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00"}
Jul 02 13:59:35.451886 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.451861    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"
Jul 02 13:59:35.452319 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.451898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf" event={"ID":"8d70523c-c9be-4ba9-baa8-29ec77d3bd0f","Type":"ContainerDied","Data":"a5439825c4e5c55e008c0417832dee6e9411c32059ed9da4aeb3b41f7b7a5889"}
Jul 02 13:59:35.452319 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.451922    2564 scope.go:117] "RemoveContainer" containerID="f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00"
Jul 02 13:59:35.461479 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.461462    2564 scope.go:117] "RemoveContainer" containerID="faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7"
Jul 02 13:59:35.469147 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.469131    2564 scope.go:117] "RemoveContainer" containerID="f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00"
Jul 02 13:59:35.469394 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:59:35.469376    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00\": container with ID starting with f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00 not found: ID does not exist" containerID="f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00"
Jul 02 13:59:35.469449 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.469404    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00"} err="failed to get container status \"f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00\": rpc error: code = NotFound desc = could not find container \"f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00\": container with ID starting with f48f76267085613c1faabdeab8a65e24d4928b8470769b70845ae8f50ae63f00 not found: ID does not exist"
Jul 02 13:59:35.469449 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.469421    2564 scope.go:117] "RemoveContainer" containerID="faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7"
Jul 02 13:59:35.469707 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:59:35.469690    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7\": container with ID starting with faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7 not found: ID does not exist" containerID="faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7"
Jul 02 13:59:35.469767 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.469714    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7"} err="failed to get container status \"faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7\": rpc error: code = NotFound desc = could not find container \"faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7\": container with ID starting with faa91271e481b84e9f280fc41ad01ac5638ee5c3cc6a258974f3e2eab6c09ca7 not found: ID does not exist"
Jul 02 13:59:35.474485 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.474464    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"]
Jul 02 13:59:35.478023 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.478003    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf"]
Jul 02 13:59:35.761914 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:35.761849    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" path="/var/lib/kubelet/pods/8d70523c-c9be-4ba9-baa8-29ec77d3bd0f/volumes"
Jul 02 13:59:55.912825 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:59:55.912782    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 13:59:55.912825 ip-10-0-138-135 kubenswrapper[2564]: E0702 13:59:55.912791    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 13:59:56.554581 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:56.554550    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 13:59:56.554758 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:56.554555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 13:59:57.045995 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.045961    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 02 13:59:57.046378 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.046365    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="storage-initializer"
Jul 02 13:59:57.046422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.046379    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="storage-initializer"
Jul 02 13:59:57.046422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.046393    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main"
Jul 02 13:59:57.046422 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.046399    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main"
Jul 02 13:59:57.046519 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.046462    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d70523c-c9be-4ba9-baa8-29ec77d3bd0f" containerName="main"
Jul 02 13:59:57.049512 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.049494    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.052214 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.052196    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:59:57.052338 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.052236    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 02 13:59:57.052338 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.052251    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dqt4b\""
Jul 02 13:59:57.064240 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.064217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 02 13:59:57.074930 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.074908    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.075041 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.074939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d2f884-bd67-4fa9-812a-56053c2ab869-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.075041 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.074958    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.075041 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.074980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.075180 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.075096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7rp\" (UniqueName: \"kubernetes.io/projected/18d2f884-bd67-4fa9-812a-56053c2ab869-kube-api-access-xd7rp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.075180 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.075147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.075180 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.075173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176188 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176336 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d2f884-bd67-4fa9-812a-56053c2ab869-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176336 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176336 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176336 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176263    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7rp\" (UniqueName: \"kubernetes.io/projected/18d2f884-bd67-4fa9-812a-56053c2ab869-kube-api-access-xd7rp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176336 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176336 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176693 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176595    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.176930 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176828    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.177070 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176896    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.177070 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.176964    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.179351 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.179116    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d2f884-bd67-4fa9-812a-56053c2ab869-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.179559 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.179439    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.184226 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.184206    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7rp\" (UniqueName: \"kubernetes.io/projected/18d2f884-bd67-4fa9-812a-56053c2ab869-kube-api-access-xd7rp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.362756 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.362684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 13:59:57.500915 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.500745    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 02 13:59:57.503586 ip-10-0-138-135 kubenswrapper[2564]: W0702 13:59:57.503545    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d2f884_bd67_4fa9_812a_56053c2ab869.slice/crio-ecd66c686c7e15a2e1060f58b8c8394afd0c565ed9c7c364c75bab4945275a09 WatchSource:0}: Error finding container ecd66c686c7e15a2e1060f58b8c8394afd0c565ed9c7c364c75bab4945275a09: Status 404 returned error can't find the container with id ecd66c686c7e15a2e1060f58b8c8394afd0c565ed9c7c364c75bab4945275a09
Jul 02 13:59:57.563415 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:57.563387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"18d2f884-bd67-4fa9-812a-56053c2ab869","Type":"ContainerStarted","Data":"ecd66c686c7e15a2e1060f58b8c8394afd0c565ed9c7c364c75bab4945275a09"}
Jul 02 13:59:58.575156 ip-10-0-138-135 kubenswrapper[2564]: I0702 13:59:58.575107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"18d2f884-bd67-4fa9-812a-56053c2ab869","Type":"ContainerStarted","Data":"00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b"}
Jul 02 14:00:01.600264 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:01.600181    2564 generic.go:356] "Generic (PLEG): container finished" podID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerID="00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b" exitCode=0
Jul 02 14:00:01.600633 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:01.600254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"18d2f884-bd67-4fa9-812a-56053c2ab869","Type":"ContainerDied","Data":"00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b"}
Jul 02 14:00:02.607598 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:02.607565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"18d2f884-bd67-4fa9-812a-56053c2ab869","Type":"ContainerStarted","Data":"5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8"}
Jul 02 14:00:02.630371 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:02.630313    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.856503677 podStartE2EDuration="5.63029285s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773789173s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:00:02.62782922 +0000 UTC m=+2777.358867390" watchObservedRunningTime="2026-07-02 14:00:02.63029285 +0000 UTC m=+2777.361330984"
Jul 02 14:00:03.363809 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:03.363768    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 14:00:07.363829 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:07.363791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 14:00:07.365421 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:07.365392    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:00:17.363277 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:17.363187    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:00:27.363080 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:27.363041    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:00:32.390697 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:00:32.390654    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:00:32.449694 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:00:32.449657    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:00:32.767266 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:32.767178    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:00:32.767907 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:32.767886    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:00:37.363916 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:37.363872    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:00:47.363591 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:47.363548    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:00:57.364054 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:00:57.364006    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:01:07.363187 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:01:07.363144    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:01:17.363644 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:01:17.363594    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:01:27.363952 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:01:27.363904    2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 02 14:01:37.372906 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:01:37.372870    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 14:01:37.380264 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:01:37.380240    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 14:01:59.555295 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:01:59.555241    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:01:59.555295 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:01:59.555248    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:02:00.227894 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:02:00.227863    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:02:00.228058 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:02:00.227868    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:02:35.768607 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:02:35.768571    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:02:35.769075 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:02:35.768923    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:02:36.406794 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:02:36.406763    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:02:36.406992 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:02:36.406763    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:03:46.028913 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:03:46.028837    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:03:46.040974 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:03:46.040953    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:04:03.228710 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:04:03.228669    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:04:03.228710 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:04:03.228673    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:04:03.850041 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:04:03.850010    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:04:03.850224 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:04:03.850014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:04:39.407688 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:04:39.407586    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:04:39.407688 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:04:39.407599    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:04:40.025336 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:04:40.025305    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:04:40.025568 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:04:40.025348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:06:06.851467 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:06:06.851420    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:06:06.851467 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:06:06.851421    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:06:07.468975 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:06:07.468933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:06:07.469184 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:06:07.468933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:06:43.028579 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:06:43.028460    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:06:43.029678 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:06:43.029645    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:06:43.651246 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:06:43.651214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:06:43.651445 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:06:43.651214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:08:10.469539 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:08:10.469482    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:08:10.470026 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:08:10.469794    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:08:11.088991 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:08:11.088957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:08:11.089205 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:08:11.088957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:08:46.062291 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:08:46.062258    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:08:46.076229 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:08:46.076197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:08:46.652312 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:08:46.652261    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:08:46.652312 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:08:46.652266    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:08:47.277895 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:08:47.277865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:08:47.278370 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:08:47.277908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:10:14.090057 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:10:14.090004    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:10:14.090057 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:10:14.090004    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:10:14.733221 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:10:14.733185    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:10:14.733435 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:10:14.733188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:10:50.278841 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:10:50.278758    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:10:50.278841 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:10:50.278781    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:10:50.911472 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:10:50.911436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:10:50.911686 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:10:50.911436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:12:17.734366 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:12:17.734285    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:12:17.734366 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:12:17.734310    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:12:18.345014 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:12:18.344984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:12:18.345180 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:12:18.344989    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:12:53.912195 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:12:53.912155    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:12:53.912821 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:12:53.912215    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:12:54.518021 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:12:54.517994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:12:54.518175 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:12:54.517997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:13:46.094822 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:13:46.094794    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:13:46.109476 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:13:46.109451    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:14:21.345865 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:14:21.345814    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:14:21.346361 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:14:21.345866    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:14:21.967541 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:14:21.967484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:14:21.967769 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:14:21.967486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:14:57.519666 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:14:57.519600    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:14:57.519666 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:14:57.519600    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:14:58.165764 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:14:58.165728    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:14:58.165954 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:14:58.165728    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:15:29.398635 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:29.398552    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 02 14:15:44.422779 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:44.422720    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main" containerID="cri-o://5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8" gracePeriod=45
Jul 02 14:15:45.194187 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.194158    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 14:15:45.231727 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-tmp-dir\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.231727 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231720    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d2f884-bd67-4fa9-812a-56053c2ab869-tls-certs\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.231949 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231763    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd7rp\" (UniqueName: \"kubernetes.io/projected/18d2f884-bd67-4fa9-812a-56053c2ab869-kube-api-access-xd7rp\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.231949 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231787    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-dshm\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.231949 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231819    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-kserve-provision-location\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.231949 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-model-cache\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.231949 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.231921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-home\") pod \"18d2f884-bd67-4fa9-812a-56053c2ab869\" (UID: \"18d2f884-bd67-4fa9-812a-56053c2ab869\") "
Jul 02 14:15:45.232873 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.232824    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-home" (OuterVolumeSpecName: "home") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:45.233173 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.232958    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-model-cache" (OuterVolumeSpecName: "model-cache") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:45.234634 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.234610    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d2f884-bd67-4fa9-812a-56053c2ab869-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:15:45.234785 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.234728    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d2f884-bd67-4fa9-812a-56053c2ab869-kube-api-access-xd7rp" (OuterVolumeSpecName: "kube-api-access-xd7rp") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "kube-api-access-xd7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:15:45.234785 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.234734    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-dshm" (OuterVolumeSpecName: "dshm") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:45.245113 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.245078    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:45.305117 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.305068    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18d2f884-bd67-4fa9-812a-56053c2ab869" (UID: "18d2f884-bd67-4fa9-812a-56053c2ab869"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:45.333483 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333455    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d2f884-bd67-4fa9-812a-56053c2ab869-tls-certs\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.333483 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333480    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7rp\" (UniqueName: \"kubernetes.io/projected/18d2f884-bd67-4fa9-812a-56053c2ab869-kube-api-access-xd7rp\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.333483 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333489    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-dshm\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.333733 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333498    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-kserve-provision-location\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.333733 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333507    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-model-cache\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.333733 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333515    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-home\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.333733 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.333522    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d2f884-bd67-4fa9-812a-56053c2ab869-tmp-dir\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:15:45.417124 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.417090    2564 generic.go:356] "Generic (PLEG): container finished" podID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerID="5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8" exitCode=0
Jul 02 14:15:45.417310 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.417145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"18d2f884-bd67-4fa9-812a-56053c2ab869","Type":"ContainerDied","Data":"5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8"}
Jul 02 14:15:45.417310 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.417204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"18d2f884-bd67-4fa9-812a-56053c2ab869","Type":"ContainerDied","Data":"ecd66c686c7e15a2e1060f58b8c8394afd0c565ed9c7c364c75bab4945275a09"}
Jul 02 14:15:45.417310 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.417153    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 02 14:15:45.417310 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.417229    2564 scope.go:117] "RemoveContainer" containerID="5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8"
Jul 02 14:15:45.427057 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.426828    2564 scope.go:117] "RemoveContainer" containerID="00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b"
Jul 02 14:15:45.435306 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.435291    2564 scope.go:117] "RemoveContainer" containerID="5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8"
Jul 02 14:15:45.435580 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:15:45.435563    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8\": container with ID starting with 5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8 not found: ID does not exist" containerID="5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8"
Jul 02 14:15:45.435638 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.435588    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8"} err="failed to get container status \"5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8\": rpc error: code = NotFound desc = could not find container \"5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8\": container with ID starting with 5215bae452a54bf72f6b814d8faa2bfddaf01cf9a12a1d13eaa832d7159e18c8 not found: ID does not exist"
Jul 02 14:15:45.435638 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.435603    2564 scope.go:117] "RemoveContainer" containerID="00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b"
Jul 02 14:15:45.435831 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:15:45.435815    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b\": container with ID starting with 00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b not found: ID does not exist" containerID="00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b"
Jul 02 14:15:45.435873 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.435835    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b"} err="failed to get container status \"00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b\": rpc error: code = NotFound desc = could not find container \"00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b\": container with ID starting with 00bc2a8220579f6aa16c22670962294b6413680f4d43866064950d2f03aafd8b not found: ID does not exist"
Jul 02 14:15:45.442029 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.442007    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 02 14:15:45.447242 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.447222    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 02 14:15:45.761360 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:15:45.761328    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" path="/var/lib/kubelet/pods/18d2f884-bd67-4fa9-812a-56053c2ab869/volumes"
Jul 02 14:16:24.968207 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:16:24.968166    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:16:24.968207 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:16:24.968166    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:16:25.622345 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:16:25.622311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:16:25.622555 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:16:25.622311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:17:01.167247 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:17:01.167208    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:17:01.167247 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:17:01.167206    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:17:01.796678 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:17:01.796645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:17:01.796846 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:17:01.796645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:18:28.623588 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:18:28.623476    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:18:28.623588 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:18:28.623476    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:18:29.220192 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:18:29.220146    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:18:29.220382 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:18:29.220146    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:18:46.124335 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:18:46.124310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:18:46.140945 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:18:46.140919    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:19:04.797156 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:19:04.797108    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:19:04.797579 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:19:04.797175    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:19:05.394012 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:19:05.393976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:19:05.394222 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:19:05.393976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:20:32.221608 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:20:32.221561    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:20:32.221608 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:20:32.221583    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:20:32.824754 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:20:32.824721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:20:32.824938 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:20:32.824729    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:21:08.395675 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:21:08.395591    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:21:08.395675 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:21:08.395591    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:21:09.007434 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:21:09.007397    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:21:09.007637 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:21:09.007397    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:22:35.825935 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:22:35.825876    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:22:35.825935 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:22:35.825876    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:22:36.442465 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:22:36.442434    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:22:36.442675 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:22:36.442434    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:23:12.008994 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:23:12.008938    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:23:12.008994 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:23:12.008939    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:23:12.627727 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:23:12.627684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:23:12.627905 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:23:12.627684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:23:46.156335 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:23:46.156308    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:23:46.173660 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:23:46.173633    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:24:39.443190 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:24:39.443108    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:24:39.443190 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:24:39.443107    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:24:40.060118 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:24:40.060095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:24:40.060300 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:24:40.060094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:25:15.629025 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:25:15.628984    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:25:15.629025 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:25:15.629001    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:25:16.232746 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:25:16.232712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:25:16.232958 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:25:16.232713    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:26:43.060716 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:26:43.060678    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:26:43.060716 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:26:43.060675    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:26:43.668816 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:26:43.668784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:26:43.668816 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:26:43.668808    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:27:19.234163 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:27:19.234075    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:27:19.234769 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:27:19.234150    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:27:19.836783 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:27:19.836749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:27:19.836945 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:27:19.836754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:28:46.184376 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:28:46.184314    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:28:46.203691 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:28:46.203674    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:28:46.669688 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:28:46.669656    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:28:46.669688 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:28:46.669679    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:28:47.261587 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:28:47.261557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:28:47.261932 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:28:47.261557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:29:22.838110 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:29:22.838060    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:29:22.838110 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:29:22.838101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:29:23.438584 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:29:23.438561    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:29:23.438756 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:29:23.438561    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:30:50.262943 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:30:50.262862    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:30:50.262943 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:30:50.262871    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:30:50.857363 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:30:50.857337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:30:50.857564 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:30:50.857337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:31:26.439961 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:31:26.439920    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:31:26.439961 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:31:26.439948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:31:27.020738 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:31:27.020705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:31:27.020903 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:31:27.020707    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:32:53.858648 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:32:53.858600    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:32:53.858648 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:32:53.858635    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:32:54.445946 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:32:54.445913    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:32:54.446112 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:32:54.445913    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:33:30.021877 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:33:30.021768    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:33:30.021877 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:33:30.021768    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:33:30.622979 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:33:30.622943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:33:30.622979 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:33:30.622976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:33:46.215379 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:33:46.215262    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:33:46.236413 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:33:46.236389    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:34:46.933487 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:34:46.933393    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-6bbd58757b-88495"]
Jul 02 14:34:46.934080 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:34:46.933639    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495" podUID="38758751-9e67-4e38-847f-e0c26baa24dd" containerName="manager" containerID="cri-o://e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df" gracePeriod=30
Jul 02 14:34:57.447120 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:34:57.447083    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:34:57.447578 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:34:57.447176    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:34:58.049225 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:34:58.049192    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:34:58.049402 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:34:58.049192    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:35:03.481441 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.481415    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 14:35:03.548914 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.548880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38758751-9e67-4e38-847f-e0c26baa24dd-cert\") pod \"38758751-9e67-4e38-847f-e0c26baa24dd\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") "
Jul 02 14:35:03.549072 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.548940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rvd7\" (UniqueName: \"kubernetes.io/projected/38758751-9e67-4e38-847f-e0c26baa24dd-kube-api-access-4rvd7\") pod \"38758751-9e67-4e38-847f-e0c26baa24dd\" (UID: \"38758751-9e67-4e38-847f-e0c26baa24dd\") "
Jul 02 14:35:03.550954 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.550920    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38758751-9e67-4e38-847f-e0c26baa24dd-kube-api-access-4rvd7" (OuterVolumeSpecName: "kube-api-access-4rvd7") pod "38758751-9e67-4e38-847f-e0c26baa24dd" (UID: "38758751-9e67-4e38-847f-e0c26baa24dd"). InnerVolumeSpecName "kube-api-access-4rvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:35:03.551241 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.551218    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38758751-9e67-4e38-847f-e0c26baa24dd-cert" (OuterVolumeSpecName: "cert") pod "38758751-9e67-4e38-847f-e0c26baa24dd" (UID: "38758751-9e67-4e38-847f-e0c26baa24dd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:35:03.650628 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.650603    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38758751-9e67-4e38-847f-e0c26baa24dd-cert\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.650628 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:03.650626    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvd7\" (UniqueName: \"kubernetes.io/projected/38758751-9e67-4e38-847f-e0c26baa24dd-kube-api-access-4rvd7\") on node \"ip-10-0-138-135.ec2.internal\" DevicePath \"\""
Jul 02 14:35:04.076870 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.076770    2564 generic.go:356] "Generic (PLEG): container finished" podID="38758751-9e67-4e38-847f-e0c26baa24dd" containerID="e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df" exitCode=0
Jul 02 14:35:04.076870 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.076808    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495"
Jul 02 14:35:04.076870 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.076852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495" event={"ID":"38758751-9e67-4e38-847f-e0c26baa24dd","Type":"ContainerDied","Data":"e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df"}
Jul 02 14:35:04.077139 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.076901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6bbd58757b-88495" event={"ID":"38758751-9e67-4e38-847f-e0c26baa24dd","Type":"ContainerDied","Data":"f3fe0546b85fa8d1c7ddb2a4a52886907c42646a564dda817d59c73eeeccda6d"}
Jul 02 14:35:04.077139 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.076921    2564 scope.go:117] "RemoveContainer" containerID="e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df"
Jul 02 14:35:04.088482 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.088464    2564 scope.go:117] "RemoveContainer" containerID="e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df"
Jul 02 14:35:04.088847 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:35:04.088828    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df\": container with ID starting with e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df not found: ID does not exist" containerID="e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df"
Jul 02 14:35:04.088899 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.088857    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df"} err="failed to get container status \"e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df\": rpc error: code = NotFound desc = could not find container \"e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df\": container with ID starting with e8e9ba68d2caf05aaff93c7df550bd62ce6143f66305eae0d877965eee8834df not found: ID does not exist"
Jul 02 14:35:04.094450 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.094422    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-6bbd58757b-88495"]
Jul 02 14:35:04.098483 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:04.098458    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-6bbd58757b-88495"]
Jul 02 14:35:05.761183 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:05.761153    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38758751-9e67-4e38-847f-e0c26baa24dd" path="/var/lib/kubelet/pods/38758751-9e67-4e38-847f-e0c26baa24dd/volumes"
Jul 02 14:35:33.624401 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:35:33.624361    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:35:33.624401 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:35:33.624366    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:35:34.220413 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:34.220383    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:35:34.220593 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:35:34.220383    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:37:01.050322 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:37:01.050280    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" podUID="9a61ddaa-6ef6-400a-925b-80a204a97ab0"
Jul 02 14:37:01.050322 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:37:01.050298    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" podUID="94d6d966-062a-4c8e-8461-cf08c57dddad"
Jul 02 14:37:01.643924 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:01.643889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq"
Jul 02 14:37:01.644107 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:01.643889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp"
Jul 02 14:37:37.221295 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:37:37.221256    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="d3f69700-0389-4c20-8fd5-d1d9d1608bf9"
Jul 02 14:37:37.221295 ip-10-0-138-135 kubenswrapper[2564]: E0702 14:37:37.221264    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7"
Jul 02 14:37:37.812917 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:37.812875    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 02 14:37:37.813127 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:37.812878    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 02 14:37:38.265617 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:38.265585    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:37:38.335853 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:38.335829    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:37:39.846139 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:39.846105    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:37:39.928176 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:39.928146    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:37:41.462379 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:41.462311    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:37:41.537629 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:41.537601    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:37:43.030955 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:43.030925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:37:43.106206 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:43.106178    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:37:44.169311 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:44.169285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:44.390606 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:44.390583    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:44.421639 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:44.421573    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:45.525608 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:45.525576    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:45.747965 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:45.747932    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:45.779632 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:45.779562    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:46.919199 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:46.919171    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:47.110282 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:47.110249    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:47.139945 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:47.139918    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:48.322664 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:48.322633    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:48.557386 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:48.557343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:48.588400 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:48.588332    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:49.681016 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:49.680982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:49.903717 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:49.903687    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:49.936449 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:49.936366    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:51.045924 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:51.045890    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:51.271079 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:51.271042    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:51.299875 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:51.299793    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:52.416499 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:52.416469    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:52.646550 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:52.646488    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:52.674336 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:52.674242    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:53.782478 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:53.782436    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:54.009440 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:54.009407    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:54.037900 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:54.037825    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:55.131772 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:55.131744    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:55.366073 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:55.366030    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:55.394278 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:55.394197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:56.540431 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:56.540394    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:56.768787 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:56.768746    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:56.805730 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:56.805635    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:57.943784 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:57.943752    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:58.169042 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:58.168986    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:58.195141 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:58.195052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:37:59.296563 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:59.296508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:37:59.516056 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:59.516028    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:37:59.540275 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:37:59.540251    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:00.645544 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:00.645478    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:00.866220 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:00.866187    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:00.892692 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:00.892661    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:02.013612 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:02.013584    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:02.234271 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:02.234220    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:02.262606 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:02.262577    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:03.366398 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:03.366367    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:03.590382 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:03.590343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:03.620203 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:03.620135    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:04.695248 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:04.695220    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:04.921629 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:04.921594    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:04.946386 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:04.946317    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:06.045560 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:06.045509    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:06.272865 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:06.272830    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:06.300300 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:06.300229    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:07.465147 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:07.465125    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:07.698986 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:07.698957    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:07.732705 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:07.732636    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:08.878328 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:08.878304    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:09.099958 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:09.099925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:09.147658 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:09.147591    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:10.253209 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:10.253169    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:10.440623 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:10.440597    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:10.480544 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:10.480508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:11.676142 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:11.676117    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:11.926031 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:11.926005    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:11.951685 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:11.951621    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:17.767657 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:17.767624    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kkm78_03505741-d6c7-4043-b2b0-28878d7c2a5d/global-pull-secret-syncer/0.log"
Jul 02 14:38:17.937387 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:17.937360    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wddlr_f8369f04-06c1-484f-b940-555722659bf6/konnectivity-agent/0.log"
Jul 02 14:38:18.022868 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:18.022792    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-135.ec2.internal_eb7a4c8ad16d6f669b998210c1cb6767/haproxy/0.log"
Jul 02 14:38:23.959931 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:23.959893    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-fkvcz_80ee4fa5-5c1f-48f3-8e4a-d1023ba49100/manager/0.log"
Jul 02 14:38:24.232059 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:24.231968    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_eff42df7-ea1d-460d-bfb6-5892c1cdb86e/manager/0.log"
Jul 02 14:38:24.288797 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:24.288764    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-kfnjg_2b916bad-a834-4461-a5a6-69eba9c73514/manager/0.log"
Jul 02 14:38:26.012288 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:26.012244    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbjmw_4cd67921-d031-4799-9f13-fa3c69680741/node-exporter/0.log"
Jul 02 14:38:26.035219 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:26.035196    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbjmw_4cd67921-d031-4799-9f13-fa3c69680741/kube-rbac-proxy/0.log"
Jul 02 14:38:26.060396 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:26.060376    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbjmw_4cd67921-d031-4799-9f13-fa3c69680741/init-textfile/0.log"
Jul 02 14:38:26.439047 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:26.439019    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-n48r6_0f10aaa2-ab79-4739-8fe6-709eebc22231/prometheus-operator-admission-webhook/0.log"
Jul 02 14:38:27.720271 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:27.720243    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-kbdhf_d869b75a-3b17-4a6a-ba2a-18817bad3c3e/networking-console-plugin/0.log"
Jul 02 14:38:28.127844 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.127811    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"]
Jul 02 14:38:28.128226 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128210    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38758751-9e67-4e38-847f-e0c26baa24dd" containerName="manager"
Jul 02 14:38:28.128285 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128227    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38758751-9e67-4e38-847f-e0c26baa24dd" containerName="manager"
Jul 02 14:38:28.128285 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128242    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="storage-initializer"
Jul 02 14:38:28.128285 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128247    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="storage-initializer"
Jul 02 14:38:28.128285 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128254    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main"
Jul 02 14:38:28.128285 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128260    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main"
Jul 02 14:38:28.128564 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128318    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="38758751-9e67-4e38-847f-e0c26baa24dd" containerName="manager"
Jul 02 14:38:28.128564 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.128327    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d2f884-bd67-4fa9-812a-56053c2ab869" containerName="main"
Jul 02 14:38:28.131629 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.131605    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.135306 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.135285    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfr7l\"/\"kube-root-ca.crt\""
Jul 02 14:38:28.136465 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.136450    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfr7l\"/\"default-dockercfg-cvgg4\""
Jul 02 14:38:28.136557 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.136461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfr7l\"/\"openshift-service-ca.crt\""
Jul 02 14:38:28.143575 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.143553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"]
Jul 02 14:38:28.229158 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.229121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vw6\" (UniqueName: \"kubernetes.io/projected/5563b21a-3591-4610-81b0-8e62b346c95f-kube-api-access-27vw6\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.229331 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.229168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-proc\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.229331 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.229195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-podres\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.229331 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.229266    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-lib-modules\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.229331 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.229299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-sys\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330386 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vw6\" (UniqueName: \"kubernetes.io/projected/5563b21a-3591-4610-81b0-8e62b346c95f-kube-api-access-27vw6\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330600 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-proc\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330600 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330472    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-proc\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330600 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-podres\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330736 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-lib-modules\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330736 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-sys\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330736 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330701    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-podres\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330736 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330730    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-sys\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.330911 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.330769    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5563b21a-3591-4610-81b0-8e62b346c95f-lib-modules\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.339487 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.339453    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vw6\" (UniqueName: \"kubernetes.io/projected/5563b21a-3591-4610-81b0-8e62b346c95f-kube-api-access-27vw6\") pod \"perf-node-gather-daemonset-lf4gk\" (UID: \"5563b21a-3591-4610-81b0-8e62b346c95f\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.441360 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.441262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:28.573147 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.573117    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"]
Jul 02 14:38:28.575735 ip-10-0-138-135 kubenswrapper[2564]: W0702 14:38:28.575703    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5563b21a_3591_4610_81b0_8e62b346c95f.slice/crio-8d498b59e8a6b662d6546d3418ded734b2859a1ff1bcdb385e8b3d83a7a42db4 WatchSource:0}: Error finding container 8d498b59e8a6b662d6546d3418ded734b2859a1ff1bcdb385e8b3d83a7a42db4: Status 404 returned error can't find the container with id 8d498b59e8a6b662d6546d3418ded734b2859a1ff1bcdb385e8b3d83a7a42db4
Jul 02 14:38:28.781365 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:28.781284    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86459b6747-h2gl2_46c94572-6b06-42b4-b411-71759517e639/console/0.log"
Jul 02 14:38:29.059829 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:29.059736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk" event={"ID":"5563b21a-3591-4610-81b0-8e62b346c95f","Type":"ContainerStarted","Data":"523da821ac769fda9a49d8724779f190284d97773899e3862fd97426bc231263"}
Jul 02 14:38:29.059829 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:29.059790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk" event={"ID":"5563b21a-3591-4610-81b0-8e62b346c95f","Type":"ContainerStarted","Data":"8d498b59e8a6b662d6546d3418ded734b2859a1ff1bcdb385e8b3d83a7a42db4"}
Jul 02 14:38:29.060071 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:29.059842    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:29.077054 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:29.077008    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk" podStartSLOduration=1.076995529 podStartE2EDuration="1.076995529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:29.076625385 +0000 UTC m=+5083.807663538" watchObservedRunningTime="2026-07-02 14:38:29.076995529 +0000 UTC m=+5083.808033657"
Jul 02 14:38:30.674214 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:30.674178    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj9zq_10094788-5376-4e89-b777-f4a84aee36e7/dns/0.log"
Jul 02 14:38:30.695223 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:30.695201    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj9zq_10094788-5376-4e89-b777-f4a84aee36e7/kube-rbac-proxy/0.log"
Jul 02 14:38:30.717385 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:30.717364    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8qzd5_1554f3e5-e6ba-4f9e-8211-c43c57869d19/dns-node-resolver/0.log"
Jul 02 14:38:31.391149 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:31.391122    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-klrnt_e7133a6f-870e-4461-9c40-37e238ac59a2/node-ca/0.log"
Jul 02 14:38:32.920255 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:32.920227    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:38:33.008422 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:33.008390    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:38:33.570797 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:33.570770    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnxg6_a04ed4c6-8a71-41a5-b4d6-ab56ca5a68c4/serve-healthcheck-canary/0.log"
Jul 02 14:38:34.211711 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:34.211675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pwm2v_0486cd46-5551-4638-a225-4925d0eaf8e4/kube-rbac-proxy/0.log"
Jul 02 14:38:34.233050 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:34.233027    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pwm2v_0486cd46-5551-4638-a225-4925d0eaf8e4/exporter/0.log"
Jul 02 14:38:34.257969 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:34.257936    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pwm2v_0486cd46-5551-4638-a225-4925d0eaf8e4/extractor/0.log"
Jul 02 14:38:35.073465 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:35.073436    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-lf4gk"
Jul 02 14:38:38.013347 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:38.013316    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-694f5b7594-pmlpc_d0f8a0bd-838c-4389-b512-2a05e88d0803/manager/0.log"
Jul 02 14:38:38.998795 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:38.998761    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-zzskt_9dd94d58-1c1c-45f6-a911-1b879fc05b86/manager/0.log"
Jul 02 14:38:40.009808 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:40.009729    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:38:40.084623 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:40.084596    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:38:41.676805 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:41.676771    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68rqc_23b14cdb-03b2-444e-9412-2db9b25c1614/discovery/0.log"
Jul 02 14:38:41.764328 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:41.764295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77ccfbd465-p22wf_c10eca5a-056c-4119-bb57-85fa2cbe1714/router/0.log"
Jul 02 14:38:43.931344 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:43.931296    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-xdn7b_1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb/migrator/0.log"
Jul 02 14:38:43.951758 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:43.951729    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-xdn7b_1ef0b6bc-e587-4ef3-8ac6-30cf3cdacbdb/graceful-termination/0.log"
Jul 02 14:38:44.340668 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:44.340581    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-dc924_4226ad5c-5308-47fa-bb59-a4f1ad054b23/kube-storage-version-migrator-operator/1.log"
Jul 02 14:38:44.342217 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:44.342192    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-dc924_4226ad5c-5308-47fa-bb59-a4f1ad054b23/kube-storage-version-migrator-operator/0.log"
Jul 02 14:38:45.332605 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.332571    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-224qj_bb3278bb-67d1-4e15-a8d6-30b5b204ddcb/kube-multus/0.log"
Jul 02 14:38:45.445424 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.445396    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/kube-multus-additional-cni-plugins/0.log"
Jul 02 14:38:45.467672 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.467643    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/egress-router-binary-copy/0.log"
Jul 02 14:38:45.488352 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.488323    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/cni-plugins/0.log"
Jul 02 14:38:45.511214 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.511190    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/bond-cni-plugin/0.log"
Jul 02 14:38:45.538000 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.537973    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/routeoverride-cni/0.log"
Jul 02 14:38:45.559871 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.559847    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/whereabouts-cni-bincopy/0.log"
Jul 02 14:38:45.583637 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:45.583561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bdbq_908c8eb7-5b2b-49bd-8952-b68e85db9c4c/whereabouts-cni/0.log"
Jul 02 14:38:46.008621 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:46.008586    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q86fn_f11610b8-2546-4823-88c3-8d6564b1aac8/network-metrics-daemon/0.log"
Jul 02 14:38:46.030796 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:46.030760    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q86fn_f11610b8-2546-4823-88c3-8d6564b1aac8/kube-rbac-proxy/0.log"
Jul 02 14:38:46.245032 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:46.244926    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"
Jul 02 14:38:46.267113 ip-10-0-138-135 kubenswrapper[2564]: I0702 14:38:46.267057    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cfn87_e3d20200-1c44-48fd-80ad-345e9bc3d8b5/ovn-acl-logging/0.log"