Jul 16 21:47:58.604572 ip-10-0-136-80 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:47:58.604581 ip-10-0-136-80 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:47:58.604588 ip-10-0-136-80 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:47:58.604805 ip-10-0-136-80 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:48:09.825508 ip-10-0-136-80 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:48:09.825523 ip-10-0-136-80 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b5d234487fa3461aa3342ce322ba205d --
Jul 16 21:49:48.904108 ip-10-0-136-80 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:49:49.322267 ip-10-0-136-80 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:49:49.322267 ip-10-0-136-80 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:49:49.322267 ip-10-0-136-80 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:49:49.322267 ip-10-0-136-80 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:49:49.322267 ip-10-0-136-80 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:49:49.323198 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.323122    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:49:49.329588 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329567    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:49.329588 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329584    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:49.329588 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329588    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:49.329588 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329591    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:49.329588 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329594    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329597    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329600    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329603    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329606    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329608    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329611    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329615    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329617    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329619    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329622    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329624    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329627    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329629    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329632    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329634    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329637    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329639    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329642    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:49.329771 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329650    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329653    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329656    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329658    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329660    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329663    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329665    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329668    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329670    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329673    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329675    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329677    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329680    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329682    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329686    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329693    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329696    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329699    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329702    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329705    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:49.330253 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329707    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329710    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329712    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329715    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329717    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329720    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329723    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329727    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329730    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329733    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329735    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329738    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329740    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329742    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329745    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329747    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329749    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329753    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329755    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329757    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:49.330707 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329760    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329762    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329765    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329769    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329772    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329776    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329779    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329782    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329786    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329789    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329792    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329795    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329797    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329800    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329802    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329804    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329807    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329822    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329825    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329827    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:49.331196 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329830    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329833    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329835    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329838    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329840    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329843    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329845    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329848    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329850    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329853    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329855    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329857    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329859    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329862    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329864    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.329867    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330525    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330531    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330535    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330538    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:49.331664 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330540    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330543    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330545    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330548    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330551    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330553    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330556    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330559    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330561    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330564    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330566    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330568    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330571    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330574    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330576    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330579    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330581    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330583    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330586    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330589    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:49.332212 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330593    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330597    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330600    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330603    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330606    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330609    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330611    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330615    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330617    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330620    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330624    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330627    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330629    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330632    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330634    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330638    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330640    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330643    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330645    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:49.332700 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330648    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330650    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330652    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330654    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330657    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330659    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330662    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330665    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330667    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330670    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330672    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330674    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330676    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330679    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330681    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330683    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330686    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330688    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330690    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330693    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:49.333185 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330695    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330698    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330700    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330703    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330705    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330708    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330710    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330713    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330715    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330718    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330720    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330723    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330725    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330728    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330730    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330732    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330735    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330737    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330740    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:49.333645 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330742    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330745    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330747    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330749    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330753    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330755    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330757    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330760    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330762    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330764    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330767    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330769    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330772    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330774    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330777    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330779    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.330782    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331158    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331162    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331165    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:49.334119 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331168    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331171    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331173    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331175    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331178    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331180    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331183    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331185    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331188    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331190    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331192    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331195    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331197    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331201    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331203    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331206    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331208    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331211    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331213    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331215    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:49.334573 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331218    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331221    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331223    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331225    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331228    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331231    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331233    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331235    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331238    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331240    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331243    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331245    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331248    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331250    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331253    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331255    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331257    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331261    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331264    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331267    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:49.335087 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331270    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331272    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331275    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331278    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331280    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331283    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331286    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331288    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331291    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331294    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331297    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331299    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331302    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331305    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331307    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331309    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331315    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331318    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331320    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:49.335559 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331324    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331327    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331330    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331333    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331336    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331339    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331341    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331344    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331346    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331349    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331351    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331354    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331356    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331358    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331361    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331363    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331365    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331368    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331370    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331373    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:49.336045 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331376    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331378    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331381    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331383    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331387    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331389    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331392    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331394    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331398    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331402    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331406    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331408    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331411    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331413    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331415    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331418    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.331420    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331484    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331491    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331496    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331501    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:49:49.336544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331506    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331511    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331516    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331522    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331527    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331532    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331536    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331539    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331542    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331545    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331547    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331550    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331554    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331557    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331560    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331563    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331566    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331569    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331572    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331576    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331580    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331583    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331586    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331589    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:49:49.337071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331591    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331594    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331597    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331600    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331603    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331606    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331609    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331612    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331614    2557 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331617    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331621    2557 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331624    2557 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331627    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331629    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331632    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331635    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331638    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331641    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331643    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331646    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331648    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331652    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331655    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331658    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331662    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:49:49.337599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331665    2557 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331669    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331671    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331675    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331678    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331681    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331683    2557 flags.go:64] FLAG: --help="false"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331686    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331689    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331691    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331694    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331697    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331700    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331702    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331705    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331708    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331710    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331713    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331715    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331718    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331720    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331723    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331725    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331728    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:49:49.338200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331730    2557 flags.go:64] FLAG: --lock-file=""
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331733    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331735    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331738    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331743    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331746    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331749    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331753    2557 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331756    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331759    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331762    2557 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331764    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331769    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331772    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331775    2557 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331778    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331781    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331783    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331786    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331789    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331791    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331794    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331800    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331803    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331806    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:49:49.338772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331825    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331829    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331833    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331836    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331839    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331842    2557 flags.go:64] FLAG: --port="10250"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331844    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331847    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a0f2e2a95993bb91"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331850    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331853    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331863    2557 flags.go:64] FLAG: --register-node="true"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331866    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331870    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331873    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331877    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331880    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331883    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331886    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331889    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331892    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331914    2557 flags.go:64] FLAG: --runonce="false"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331917    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331919    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331922    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331925    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:49:49.339376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331927    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331930    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331933    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331936    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331938    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331941    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331944    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331946    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331949    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331952    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331954    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331959    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331962    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331965    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331968    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331971    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331973    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331976    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331978    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331981    2557 flags.go:64] FLAG: --v="2"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331985    2557 flags.go:64] FLAG: --version="false"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331990    2557 flags.go:64] FLAG: --vmodule=""
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331994    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.331997    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332105    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:49.340044 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332108    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332110    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332113    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332115    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332118    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332120    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332123    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332125    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332128    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332130    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332133    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332135    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332138    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332140    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332143    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332145    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332148    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332150    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332152    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332155    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:49.340628 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332157    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332159    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332162    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332164    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332166    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332169    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332171    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332174    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332178    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332182    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332185    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332187    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332189    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332194    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332197    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332199    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332202    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332204    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332207    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:49.341153 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332210    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332214    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332217    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332220    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332223    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332226    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332228    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332231    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332233    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332236    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332238    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332241    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332243    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332246    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332248    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332251    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332254    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332257    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332260    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:49.341604 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332262    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332264    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332268    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332272    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332275    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332278    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332280    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332283    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332285    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332288    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332290    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332293    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332296    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332298    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332300    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332302    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332305    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332307    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332310    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:49.342069 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332312    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332315    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332317    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332319    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332322    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332324    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332327    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332329    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332331    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332334    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332336    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332339    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332341    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332343    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332346    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332350    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332354    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332357    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332361    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332364    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:49.342513 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.332367    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.333037    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.339967    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.339986    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.340002    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.340008    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340069    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340074    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340077    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340080    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340083    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340086    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340090    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340094    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:49.342990 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340097    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340100    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340103    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340106    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340108    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340111    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340113    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340116    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340118    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340121    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340123    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340126    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340128    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340131    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340133    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340136    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340138    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340141    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340143    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340146    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:49.343330 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340148    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340157    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340161    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340163    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340166    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340169    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340171    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340173    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340176    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340178    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340181    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340183    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340185    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340188    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340191    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340194    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340198    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340201    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340203    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:49.343785 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340206    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340208    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340210    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340214    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340217    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340220    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340223    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340226    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340229    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340231    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340234    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340237    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340240    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340242    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340246    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340253    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340256    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340259    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340261    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:49.344290 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340264    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340266    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340269    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340271    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340274    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340276    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340278    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340281    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340283    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340286    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340288    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340290    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340293    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340295    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340298    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340300    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340303    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340305    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340307    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340310    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:49.344750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340312    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340315    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340317    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340319    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340322    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340324    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340326    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340330    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340332    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340340    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340343    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340346    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340348    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.340353    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340491    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:49.345242 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340494    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340497    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340500    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340504    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340507    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340510    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340513    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340516    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340518    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340521    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340523    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340526    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340528    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340531    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340533    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340535    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340538    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340540    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340543    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340545    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:49.345593 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340548    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340550    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340552    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340555    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340557    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340560    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340562    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340576    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340579    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340581    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340584    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340586    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340588    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340591    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340594    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340598    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340600    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340603    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340605    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:49.346067 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340607    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340610    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340612    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340615    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340617    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340619    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340622    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340624    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340626    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340629    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340631    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340635    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340638    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340641    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340644    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340646    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340649    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340651    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340654    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340656    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:49.346508 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340659    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340668    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340671    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340673    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340675    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340678    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340680    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340682    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340685    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340687    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340689    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340692    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340694    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340696    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340699    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340701    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340704    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340706    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340708    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340711    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:49.346964 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340713    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340715    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340718    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340720    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340723    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340725    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340728    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340730    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340732    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340735    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340737    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340740    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340742    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340745    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340753    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340755    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340758    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340760    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:49.347423 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:49.340762    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.340767    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.340891    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.342928    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.344390    2557 server.go:1019] "Starting client certificate rotation"
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.344491    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.344553    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:49:49.347919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.347524    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:49:49.352919 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.352845    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:49:49.362009 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.361984    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:49:49.373243 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.373223    2557 log.go:25] "Validated CRI v1 image API"
Jul 16 21:49:49.374610 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.374595    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:49:49.377383 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.377363    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ce18735-b581-41f8-bb1d-ff6e7e4d41bf:/dev/nvme0n1p4 8b8c3d7e-622a-4255-87a5-7244fa423d45:/dev/nvme0n1p3]
Jul 16 21:49:49.377452 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.377383    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:49:49.382339 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.382227    2557 manager.go:216] Machine: {Timestamp:2026-07-16 21:49:49.38093225 +0000 UTC m=+0.360281540 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3158057 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a554c1129765616f79bc836f869ce SystemUUID:ec2a554c-1129-7656-16f7-9bc836f869ce BootID:b5d23448-7fa3-461a-a334-2ce322ba205d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:91:b9:13:22:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:91:b9:13:22:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:3c:72:2c:0c:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:49:49.382339 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.382328    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:49:49.382456 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.382434    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:49:49.383629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.383613    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:49:49.385026 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.385000    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:49:49.385168 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.385031    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:49:49.385726 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.385717    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:49:49.385759 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.385728    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:49:49.385759 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.385736    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:49:49.386277 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.386265    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:49:49.387607 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.387598    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:49:49.390568 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390557    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:49:49.390681 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390672    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:49:49.390722 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390700    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:49:49.390722 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390710    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:49:49.390722 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390720    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:49:49.390856 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390728    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:49:49.390856 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.390737    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:49:49.391237 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.391222    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:49:49.391304 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.391248    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:49:49.393592 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.393569    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:49:49.395496 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.395483    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:49:49.395546 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.395513    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:49:49.396739 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396727    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396743    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396749    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396755    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396764    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396769    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396777    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:49:49.396785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396783    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:49:49.397023 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396806    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:49:49.397023 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396828    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:49:49.397023 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396844    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:49:49.397023 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.396850    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:49:49.398452 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.398438    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:49:49.398489 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.398456    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:49:49.401778 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.401763    2557 server.go:1268] "Started kubelet"
Jul 16 21:49:49.401889 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.401861    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:49:49.401950 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.401920    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:49:49.401997 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.401962    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:49:49.406659 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.405498    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:49:49.405641 ip-10-0-136-80 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:49:49.407242 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.407225    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:49:49.408622 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.408590    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:49:49.408704 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.408614    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:49:49.409302 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.409285    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:49:49.409511 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.409453    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:49:49.413876 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.413008    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-80.ec2.internal.18c2e38b33d488e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-80.ec2.internal,UID:ip-10-0-136-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-80.ec2.internal,},FirstTimestamp:2026-07-16 21:49:49.401663716 +0000 UTC m=+0.381013007,LastTimestamp:2026-07-16 21:49:49.401663716 +0000 UTC m=+0.381013007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-80.ec2.internal,}"
Jul 16 21:49:49.414473 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.414460    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:49:49.414473 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.414467    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:49:49.414584 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.414529    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.414996 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.414975    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:49:49.414996 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.414991    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:49:49.415111 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.414974    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:49:49.415153 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415126    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:49:49.415153 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415134    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:49:49.415439 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415420    2557 factory.go:55] Registering systemd factory
Jul 16 21:49:49.415536 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415467    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:49:49.415711 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415685    2557 factory.go:153] Registering CRI-O factory
Jul 16 21:49:49.415711 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415700    2557 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:49:49.415876 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415749    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:49:49.415958 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415924    2557 factory.go:103] Registering Raw factory
Jul 16 21:49:49.415958 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.415944    2557 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:49:49.416264 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.416241    2557 manager.go:349] Starting recovery of all containers
Jul 16 21:49:49.416834 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.416787    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:49:49.416925 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.416872    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:49:49.427112 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.427097    2557 manager.go:354] Recovery completed
Jul 16 21:49:49.432070 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.432058    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.433864 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.433840    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.433926 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.433876    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.433926 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.433890    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.434358 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.434341    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:49:49.434358 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.434357    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:49:49.434491 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.434373    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:49:49.434491 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.434357    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.435104 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.435088    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.435164 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.435117    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.435164 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.435127    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.435486 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.435475    2557 policy_none.go:49] "None policy: Start"
Jul 16 21:49:49.435519 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.435490    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:49:49.435519 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.435500    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:49:49.436417 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.436407    2557 policy_none.go:47] "Start"
Jul 16 21:49:49.436467 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.436399    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-80.ec2.internal.18c2e38b35bfdae0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-80.ec2.internal,UID:ip-10-0-136-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-80.ec2.internal,},FirstTimestamp:2026-07-16 21:49:49.43386288 +0000 UTC m=+0.413212169,LastTimestamp:2026-07-16 21:49:49.43386288 +0000 UTC m=+0.413212169,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-80.ec2.internal,}"
Jul 16 21:49:49.446497 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.446435    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-80.ec2.internal.18c2e38b35c0293c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-80.ec2.internal,UID:ip-10-0-136-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-80.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-80.ec2.internal,},FirstTimestamp:2026-07-16 21:49:49.43388294 +0000 UTC m=+0.413232230,LastTimestamp:2026-07-16 21:49:49.43388294 +0000 UTC m=+0.413232230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-80.ec2.internal,}"
Jul 16 21:49:49.455425 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.455238    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-80.ec2.internal.18c2e38b35c0528c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-80.ec2.internal,UID:ip-10-0-136-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-80.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-80.ec2.internal,},FirstTimestamp:2026-07-16 21:49:49.433893516 +0000 UTC m=+0.413242806,LastTimestamp:2026-07-16 21:49:49.433893516 +0000 UTC m=+0.413242806,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-80.ec2.internal,}"
Jul 16 21:49:49.462591 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.462567    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dm8rj"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.469120    2557 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-80.ec2.internal.18c2e38b35bfdae0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-80.ec2.internal.18c2e38b35bfdae0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-80.ec2.internal,UID:ip-10-0-136-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-80.ec2.internal,},FirstTimestamp:2026-07-16 21:49:49.43386288 +0000 UTC m=+0.413212169,LastTimestamp:2026-07-16 21:49:49.435103458 +0000 UTC m=+0.414452747,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-80.ec2.internal,}"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.471290    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dm8rj"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474425    2557 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.474450    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474459    2557 server.go:87] "Starting device plugin registration server"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474672    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474681    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474785    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474881    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.474890    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.475317    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:49:49.477929 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.475367    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.554629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.554590    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:49:49.555939 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.555921    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:49:49.555990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.555972    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:49:49.556036 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.555999    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:49:49.556080 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.556041    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:49:49.558601 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.558585    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:49.575693 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.575630    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.576789 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.576774    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.576851 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.576804    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.576851 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.576830    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.576949 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.576853    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.584106 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.584089    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.584150 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.584111    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-80.ec2.internal\": node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.617827 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.617790    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.656723 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.656684    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"]
Jul 16 21:49:49.656891 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.656784    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.657781 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.657765    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.657861 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.657796    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.657861 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.657806    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.659096 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659083    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.659209 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659196    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.659242 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659231    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.659766 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659749    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.659854 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659773    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.659854 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659785    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.659854 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659749    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.660019 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659880    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.660019 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.659915    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.661009 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.660994    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.661114 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.661019    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:49.661659 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.661638    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:49.661741 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.661670    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:49.661741 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.661682    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:49.681483 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.681463    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-80.ec2.internal\" not found" node="ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.685618 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.685602    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-80.ec2.internal\" not found" node="ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.718278 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.718256    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.816450 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.816418    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa5496d3168ac96273becbca4a10ee28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"fa5496d3168ac96273becbca4a10ee28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.816537 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.816452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa5496d3168ac96273becbca4a10ee28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"fa5496d3168ac96273becbca4a10ee28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.816537 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.816471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6337947e1d545ee6d7cb3a765fe8fb4c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-80.ec2.internal\" (UID: \"6337947e1d545ee6d7cb3a765fe8fb4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.818480 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.818466    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.917167 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.917138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa5496d3168ac96273becbca4a10ee28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"fa5496d3168ac96273becbca4a10ee28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.917236 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.917172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa5496d3168ac96273becbca4a10ee28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"fa5496d3168ac96273becbca4a10ee28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.917236 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.917190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6337947e1d545ee6d7cb3a765fe8fb4c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-80.ec2.internal\" (UID: \"6337947e1d545ee6d7cb3a765fe8fb4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.917314 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.917252    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa5496d3168ac96273becbca4a10ee28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"fa5496d3168ac96273becbca4a10ee28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.917314 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.917274    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa5496d3168ac96273becbca4a10ee28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"fa5496d3168ac96273becbca4a10ee28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.917314 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.917255    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6337947e1d545ee6d7cb3a765fe8fb4c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-80.ec2.internal\" (UID: \"6337947e1d545ee6d7cb3a765fe8fb4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.919247 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:49.919229    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:49.984463 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.984425    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:49.989105 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:49.988600    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:50.019339 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:50.019314    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:50.119771 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:50.119737    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:50.220263 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:50.220215    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:50.320656 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:50.320629    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:50.343798 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.343777    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:49:50.344210 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.343939    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:49:50.420901 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:50.420876    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 16 21:49:50.435697 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.435677    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:50.475876 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.475786    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:44:49 +0000 UTC" deadline="2028-02-29 22:44:48.206779311 +0000 UTC"
Jul 16 21:49:50.475876 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.475834    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14232h54m57.730949415s"
Jul 16 21:49:50.512111 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:50.512083    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6337947e1d545ee6d7cb3a765fe8fb4c.slice/crio-42ce93cf60a1893dad31bf848905e66d294a6fcb4f921a7731ef8cac01a667a8 WatchSource:0}: Error finding container 42ce93cf60a1893dad31bf848905e66d294a6fcb4f921a7731ef8cac01a667a8: Status 404 returned error can't find the container with id 42ce93cf60a1893dad31bf848905e66d294a6fcb4f921a7731ef8cac01a667a8
Jul 16 21:49:50.512485 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:50.512470    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5496d3168ac96273becbca4a10ee28.slice/crio-a39c9923f1db63e0b86df0e56cb91debaf4f4712446c5ec59adb8fb3a892fc90 WatchSource:0}: Error finding container a39c9923f1db63e0b86df0e56cb91debaf4f4712446c5ec59adb8fb3a892fc90: Status 404 returned error can't find the container with id a39c9923f1db63e0b86df0e56cb91debaf4f4712446c5ec59adb8fb3a892fc90
Jul 16 21:49:50.514757 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.514729    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:50.516726 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.516705    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:49:50.525396 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.525379    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:49:50.527340 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.527329    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 16 21:49:50.539586 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.539571    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:49:50.559957 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.559890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal" event={"ID":"6337947e1d545ee6d7cb3a765fe8fb4c","Type":"ContainerStarted","Data":"42ce93cf60a1893dad31bf848905e66d294a6fcb4f921a7731ef8cac01a667a8"}
Jul 16 21:49:50.560957 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.560938    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" event={"ID":"fa5496d3168ac96273becbca4a10ee28","Type":"ContainerStarted","Data":"a39c9923f1db63e0b86df0e56cb91debaf4f4712446c5ec59adb8fb3a892fc90"}
Jul 16 21:49:50.564124 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.564108    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:50.709077 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:50.709053    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:51.391182 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.391155    2557 apiserver.go:52] "Watching apiserver"
Jul 16 21:49:51.399079 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.399056    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:49:51.399492 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.399444    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal","openshift-dns/node-resolver-jkm6h","openshift-image-registry/node-ca-6z6pz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal","openshift-multus/multus-additional-cni-plugins-m95lw","openshift-multus/multus-hh4jx","openshift-multus/network-metrics-daemon-dpnxv","openshift-network-operator/iptables-alerter-kqgqb","kube-system/global-pull-secret-syncer-cpdkm","kube-system/konnectivity-agent-q8bwn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6","openshift-cluster-node-tuning-operator/tuned-h8jtl","openshift-network-diagnostics/network-check-target-sr4st","openshift-ovn-kubernetes/ovnkube-node-7k6b2"]
Jul 16 21:49:51.401220 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.400985    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.402281 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.402263    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:51.402365 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.402329    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:49:51.403573 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.403465    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:49:51.403706 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.403692    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gpdss\""
Jul 16 21:49:51.403769 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.403706    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:49:51.403846 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.403794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.405027 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.405014    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.405980 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.405962    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.406379 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.406361    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.406611 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.406598    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:49:51.407313 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.407163    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.407392 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.407366    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.407677 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.407660    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:49:51.407677 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.407668    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:49:51.407863 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.407850    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.408035 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.408021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7xjp4\""
Jul 16 21:49:51.408396 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.408274    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mk8fp\""
Jul 16 21:49:51.408479 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.408465    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.408538 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.408520    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:49:51.408645 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.408627    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:49:51.408751 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.408737    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:49:51.408905 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.408885    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:49:51.410106 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.409918    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.410218 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.410203    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:49:51.410554 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.410484    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.410829 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.410701    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdhdd\""
Jul 16 21:49:51.410947 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.410929    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:49:51.411143 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.411121    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:49:51.411245 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.411227    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jmtt8\""
Jul 16 21:49:51.411340 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.411285    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.411527 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.411513    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:51.411595 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.411534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.411595 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.411560    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:49:51.412019 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.412003    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.413658 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.413634    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.413993 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.413972    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.414090 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.414044    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.414758 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.414738    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:49:51.415085 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.415070    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.417019 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.417005    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:49:51.417670 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.417655    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-csbvl\""
Jul 16 21:49:51.417948 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.417934    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.418888 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.418873    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.419306 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.419293    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.419695 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.419590    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-42thw\""
Jul 16 21:49:51.419804 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.419782    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:49:51.420184 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.420167    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:49:51.420478 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.420465    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.420760 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.420726    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhhcn\""
Jul 16 21:49:51.421056 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.420954    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4gpjd\""
Jul 16 21:49:51.421106 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.421056    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:49:51.423047 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-kubelet\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423181 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-run-netns\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423181 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-node-log\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423181 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.423181 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcql\" (UniqueName: \"kubernetes.io/projected/9e266e69-5ca7-42cc-8a40-773d457c2553-kube-api-access-2tcql\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.423181 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-system-cni-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423181 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-netns\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpdr\" (UniqueName: \"kubernetes.io/projected/24fdc43a-8826-46d0-8c18-ace05937ce99-kube-api-access-cbpdr\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-os-release\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-socket-dir-parent\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-k8s-cni-cncf-io\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423283    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-multus-certs\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423303    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-ovn\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423324    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1160e2d-5f23-4b46-8369-316e2749a642-cni-binary-copy\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423364    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1160e2d-5f23-4b46-8369-316e2749a642-multus-daemon-config\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-dbus\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-sys-fs\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.423441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423437    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysconfig\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9tj\" (UniqueName: \"kubernetes.io/projected/52d062bf-25da-41c5-b1ae-0ecd872eaedf-kube-api-access-nw9tj\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-slash\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423499    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsz82\" (UniqueName: \"kubernetes.io/projected/29e8bd1d-6583-4173-9236-ab02fba76157-kube-api-access-hsz82\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52d062bf-25da-41c5-b1ae-0ecd872eaedf-host\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-cnibin\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423561    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-cnibin\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423603    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-ovnkube-config\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423683    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-socket-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzq2\" (UniqueName: \"kubernetes.io/projected/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-kube-api-access-pgzq2\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423725    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-system-cni-dir\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-os-release\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423841    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-hostroot\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.423960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-systemd\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29e8bd1d-6583-4173-9236-ab02fba76157-ovn-node-metrics-cert\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1570a9-59ea-4345-8408-5a192d1e24c6-tmp-dir\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423936    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-kubernetes\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423956    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-conf-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.423988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-etc-kubernetes\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-cni-bin\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysctl-conf\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-var-lib-kubelet\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424075    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-tuned\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-tmp\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plrnd\" (UniqueName: \"kubernetes.io/projected/35965ec5-69e1-42da-915d-2e123c7dc664-kube-api-access-plrnd\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424163    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-ovnkube-script-lib\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424186    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qn92\" (UniqueName: \"kubernetes.io/projected/fb1570a9-59ea-4345-8408-5a192d1e24c6-kube-api-access-7qn92\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424206    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52d062bf-25da-41c5-b1ae-0ecd872eaedf-serviceca\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-cni-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.424653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424252    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-cni-bin\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424272    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-kubelet\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-cni-netd\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-env-overrides\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35965ec5-69e1-42da-915d-2e123c7dc664-iptables-alerter-script\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424357    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb1570a9-59ea-4345-8408-5a192d1e24c6-hosts-file\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424399    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-registration-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-run\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj69\" (UniqueName: \"kubernetes.io/projected/e1160e2d-5f23-4b46-8369-316e2749a642-kube-api-access-6vj69\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424492    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-var-lib-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-etc-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424558    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-log-socket\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-device-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.425426 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424602    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-systemd\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-host\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-cni-binary-copy\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424683    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fba83155-ef2f-4233-a21b-155bb4abf982-agent-certs\") pod \"konnectivity-agent-q8bwn\" (UID: \"fba83155-ef2f-4233-a21b-155bb4abf982\") " pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424739    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-modprobe-d\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysctl-d\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fba83155-ef2f-4233-a21b-155bb4abf982-konnectivity-ca\") pod \"konnectivity-agent-q8bwn\" (UID: \"fba83155-ef2f-4233-a21b-155bb4abf982\") " pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-lib-modules\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424851    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6gww\" (UniqueName: \"kubernetes.io/projected/92fc2188-8ee2-47ca-98bf-a477465b096d-kube-api-access-b6gww\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-cni-multus\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424896    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-kubelet-config\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424919    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-systemd-units\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35965ec5-69e1-42da-915d-2e123c7dc664-host-slash\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.426185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.424989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-sys\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.426834 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.426248    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:49:51.432671 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.432588    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:49:51.457789 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.457758    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9xnfz"
Jul 16 21:49:51.466041 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.466024    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9xnfz"
Jul 16 21:49:51.525723 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525688    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525729    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb1570a9-59ea-4345-8408-5a192d1e24c6-hosts-file\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525751    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-registration-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-run\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj69\" (UniqueName: \"kubernetes.io/projected/e1160e2d-5f23-4b46-8369-316e2749a642-kube-api-access-6vj69\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525868    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525889    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb1570a9-59ea-4345-8408-5a192d1e24c6-hosts-file\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-registration-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-var-lib-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525967    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-etc-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525971    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-run\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.525996    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-log-socket\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-device-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-etc-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-systemd\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526122 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526044    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-log-socket\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526094    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-systemd\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526111    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-var-lib-openvswitch\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526130    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-host\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526152    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-cni-binary-copy\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-device-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526196    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-host\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fba83155-ef2f-4233-a21b-155bb4abf982-agent-certs\") pod \"konnectivity-agent-q8bwn\" (UID: \"fba83155-ef2f-4233-a21b-155bb4abf982\") " pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-modprobe-d\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysctl-d\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fba83155-ef2f-4233-a21b-155bb4abf982-konnectivity-ca\") pod \"konnectivity-agent-q8bwn\" (UID: \"fba83155-ef2f-4233-a21b-155bb4abf982\") " pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-lib-modules\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6gww\" (UniqueName: \"kubernetes.io/projected/92fc2188-8ee2-47ca-98bf-a477465b096d-kube-api-access-b6gww\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-cni-multus\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-kubelet-config\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526391    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-modprobe-d\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.526828 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.526470    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526489    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysctl-d\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.526535    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:52.026517091 +0000 UTC m=+3.005866381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526544    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-systemd-units\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35965ec5-69e1-42da-915d-2e123c7dc664-host-slash\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-sys\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-kubelet\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-cni-binary-copy\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-run-netns\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35965ec5-69e1-42da-915d-2e123c7dc664-host-slash\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526835    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-run-netns\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-sys\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-node-log\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-node-log\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.527443 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526874    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-kubelet\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526908    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-systemd-units\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fba83155-ef2f-4233-a21b-155bb4abf982-konnectivity-ca\") pod \"konnectivity-agent-q8bwn\" (UID: \"fba83155-ef2f-4233-a21b-155bb4abf982\") " pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.526992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-kubelet-config\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527011    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527016    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-cni-multus\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcql\" (UniqueName: \"kubernetes.io/projected/9e266e69-5ca7-42cc-8a40-773d457c2553-kube-api-access-2tcql\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-lib-modules\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-system-cni-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527177    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-system-cni-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527202    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-netns\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527242    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpdr\" (UniqueName: \"kubernetes.io/projected/24fdc43a-8826-46d0-8c18-ace05937ce99-kube-api-access-cbpdr\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527287    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-os-release\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527312    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-socket-dir-parent\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-k8s-cni-cncf-io\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-multus-certs\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.528202 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527409    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-multus-certs\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527411    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-k8s-cni-cncf-io\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527407    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-socket-dir-parent\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-ovn\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-os-release\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1160e2d-5f23-4b46-8369-316e2749a642-cni-binary-copy\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-ovn\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1160e2d-5f23-4b46-8369-316e2749a642-multus-daemon-config\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-dbus\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-sys-fs\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysconfig\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9tj\" (UniqueName: \"kubernetes.io/projected/52d062bf-25da-41c5-b1ae-0ecd872eaedf-kube-api-access-nw9tj\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527619    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-slash\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527632    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsz82\" (UniqueName: \"kubernetes.io/projected/29e8bd1d-6583-4173-9236-ab02fba76157-kube-api-access-hsz82\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-run-netns\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52d062bf-25da-41c5-b1ae-0ecd872eaedf-host\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.529020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527663    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysconfig\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527666    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52d062bf-25da-41c5-b1ae-0ecd872eaedf-host\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-cnibin\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-cnibin\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527715    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527733    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527754    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-ovnkube-config\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527786    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-dbus\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-socket-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-slash\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527852    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzq2\" (UniqueName: \"kubernetes.io/projected/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-kube-api-access-pgzq2\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527871    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-sys-fs\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-system-cni-dir\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-os-release\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527906    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527965    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-cnibin\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.529676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.527998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-hostroot\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-cnibin\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-systemd\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29e8bd1d-6583-4173-9236-ab02fba76157-ovn-node-metrics-cert\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-run-systemd\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1570a9-59ea-4345-8408-5a192d1e24c6-tmp-dir\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-kubernetes\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1160e2d-5f23-4b46-8369-316e2749a642-multus-daemon-config\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528097    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-hostroot\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-conf-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-conf-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528112    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1160e2d-5f23-4b46-8369-316e2749a642-cni-binary-copy\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-etc-kubernetes\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-system-cni-dir\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528180    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-etc-kubernetes\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e266e69-5ca7-42cc-8a40-773d457c2553-socket-dir\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92fc2188-8ee2-47ca-98bf-a477465b096d-os-release\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.530855 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-kubernetes\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-cni-bin\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysctl-conf\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-var-lib-kubelet\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-tuned\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528433    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-tmp\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-ovnkube-config\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plrnd\" (UniqueName: \"kubernetes.io/projected/35965ec5-69e1-42da-915d-2e123c7dc664-kube-api-access-plrnd\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1570a9-59ea-4345-8408-5a192d1e24c6-tmp-dir\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-sysctl-conf\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-ovnkube-script-lib\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528520    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-cni-bin\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qn92\" (UniqueName: \"kubernetes.io/projected/fb1570a9-59ea-4345-8408-5a192d1e24c6-kube-api-access-7qn92\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-var-lib-kubelet\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52d062bf-25da-41c5-b1ae-0ecd872eaedf-serviceca\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528578    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-cni-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-cni-bin\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.531484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-kubelet\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528652    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-cni-netd\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-env-overrides\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92fc2188-8ee2-47ca-98bf-a477465b096d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528701    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35965ec5-69e1-42da-915d-2e123c7dc664-iptables-alerter-script\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-kubelet\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-multus-cni-dir\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528869    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1160e2d-5f23-4b46-8369-316e2749a642-host-var-lib-cni-bin\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.528958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52d062bf-25da-41c5-b1ae-0ecd872eaedf-serviceca\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.529157    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29e8bd1d-6583-4173-9236-ab02fba76157-host-cni-netd\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.529195    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35965ec5-69e1-42da-915d-2e123c7dc664-iptables-alerter-script\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.529258    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.529287    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-ovnkube-script-lib\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.529299    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:52.02928471 +0000 UTC m=+3.008633986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.529651    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29e8bd1d-6583-4173-9236-ab02fba76157-env-overrides\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.530198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fba83155-ef2f-4233-a21b-155bb4abf982-agent-certs\") pod \"konnectivity-agent-q8bwn\" (UID: \"fba83155-ef2f-4233-a21b-155bb4abf982\") " pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.532174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.531322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-etc-tuned\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.532849 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.532829    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:51.532919 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.532855    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:51.532919 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.532870    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:51.532919 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:51.532915    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:49:52.032899685 +0000 UTC m=+3.012248959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:51.533056 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.532988    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-tmp\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.534943 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.534923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29e8bd1d-6583-4173-9236-ab02fba76157-ovn-node-metrics-cert\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.536778 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.536750    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj69\" (UniqueName: \"kubernetes.io/projected/e1160e2d-5f23-4b46-8369-316e2749a642-kube-api-access-6vj69\") pod \"multus-hh4jx\" (UID: \"e1160e2d-5f23-4b46-8369-316e2749a642\") " pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.541527 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.541509    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcql\" (UniqueName: \"kubernetes.io/projected/9e266e69-5ca7-42cc-8a40-773d457c2553-kube-api-access-2tcql\") pod \"aws-ebs-csi-driver-node-ctmn6\" (UID: \"9e266e69-5ca7-42cc-8a40-773d457c2553\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.541926 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.541907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzq2\" (UniqueName: \"kubernetes.io/projected/502f8f6b-bf16-49dc-bbbf-ffba51fa81c5-kube-api-access-pgzq2\") pod \"tuned-h8jtl\" (UID: \"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5\") " pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.542314 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.542297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpdr\" (UniqueName: \"kubernetes.io/projected/24fdc43a-8826-46d0-8c18-ace05937ce99-kube-api-access-cbpdr\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:51.542379 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.542322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plrnd\" (UniqueName: \"kubernetes.io/projected/35965ec5-69e1-42da-915d-2e123c7dc664-kube-api-access-plrnd\") pod \"iptables-alerter-kqgqb\" (UID: \"35965ec5-69e1-42da-915d-2e123c7dc664\") " pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.542504 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.542485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9tj\" (UniqueName: \"kubernetes.io/projected/52d062bf-25da-41c5-b1ae-0ecd872eaedf-kube-api-access-nw9tj\") pod \"node-ca-6z6pz\" (UID: \"52d062bf-25da-41c5-b1ae-0ecd872eaedf\") " pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.542718 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.542699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qn92\" (UniqueName: \"kubernetes.io/projected/fb1570a9-59ea-4345-8408-5a192d1e24c6-kube-api-access-7qn92\") pod \"node-resolver-jkm6h\" (UID: \"fb1570a9-59ea-4345-8408-5a192d1e24c6\") " pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.542797 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.542761    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6gww\" (UniqueName: \"kubernetes.io/projected/92fc2188-8ee2-47ca-98bf-a477465b096d-kube-api-access-b6gww\") pod \"multus-additional-cni-plugins-m95lw\" (UID: \"92fc2188-8ee2-47ca-98bf-a477465b096d\") " pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.544203 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.544184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsz82\" (UniqueName: \"kubernetes.io/projected/29e8bd1d-6583-4173-9236-ab02fba76157-kube-api-access-hsz82\") pod \"ovnkube-node-7k6b2\" (UID: \"29e8bd1d-6583-4173-9236-ab02fba76157\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.713582 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.713517    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:49:51.727292 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.727267    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6z6pz"
Jul 16 21:49:51.734796 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.734775    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:49:51.739939 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.739918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m95lw"
Jul 16 21:49:51.747410 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.747394    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hh4jx"
Jul 16 21:49:51.753873 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.753858    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jkm6h"
Jul 16 21:49:51.761310 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.761297    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kqgqb"
Jul 16 21:49:51.768752 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.768736    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6"
Jul 16 21:49:51.774239 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.774224    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8jtl"
Jul 16 21:49:51.924637 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:51.924610    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:52.033077 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.033005    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:52.033077 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.033047    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.033098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033183    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033189    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033196    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033207    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033221    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:52.033266 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033257    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:53.033238922 +0000 UTC m=+4.012588208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:52.033490 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033280    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:53.033265028 +0000 UTC m=+4.012614315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:52.033490 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.033299    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:49:53.03328945 +0000 UTC m=+4.012638725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:52.248443 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:52.248416    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e8bd1d_6583_4173_9236_ab02fba76157.slice/crio-aa0354193979f69b800780009b05da1156031068f04a2e92d76d00f4bf5f4951 WatchSource:0}: Error finding container aa0354193979f69b800780009b05da1156031068f04a2e92d76d00f4bf5f4951: Status 404 returned error can't find the container with id aa0354193979f69b800780009b05da1156031068f04a2e92d76d00f4bf5f4951
Jul 16 21:49:52.249657 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:52.249636    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1160e2d_5f23_4b46_8369_316e2749a642.slice/crio-845534d1b2fb6d62b11ee8cd6449ca7665ecc916ff524459d64a852705f3ef8b WatchSource:0}: Error finding container 845534d1b2fb6d62b11ee8cd6449ca7665ecc916ff524459d64a852705f3ef8b: Status 404 returned error can't find the container with id 845534d1b2fb6d62b11ee8cd6449ca7665ecc916ff524459d64a852705f3ef8b
Jul 16 21:49:52.249986 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:52.249965    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1570a9_59ea_4345_8408_5a192d1e24c6.slice/crio-edb972ea9782f277e6f567cc73d0a542b90f15360acfd4f3be4142244a46ed9a WatchSource:0}: Error finding container edb972ea9782f277e6f567cc73d0a542b90f15360acfd4f3be4142244a46ed9a: Status 404 returned error can't find the container with id edb972ea9782f277e6f567cc73d0a542b90f15360acfd4f3be4142244a46ed9a
Jul 16 21:49:52.251661 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:52.251482    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d062bf_25da_41c5_b1ae_0ecd872eaedf.slice/crio-70f35afc894ad2943e10135d8bb56ee455a5187a2aeadf829a0f57d8dd5edde6 WatchSource:0}: Error finding container 70f35afc894ad2943e10135d8bb56ee455a5187a2aeadf829a0f57d8dd5edde6: Status 404 returned error can't find the container with id 70f35afc894ad2943e10135d8bb56ee455a5187a2aeadf829a0f57d8dd5edde6
Jul 16 21:49:52.258247 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:49:52.258104    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35965ec5_69e1_42da_915d_2e123c7dc664.slice/crio-2101a458fe9577f6e0850270113b1d478a04c65da0dfad171fd0d62cd4baab7b WatchSource:0}: Error finding container 2101a458fe9577f6e0850270113b1d478a04c65da0dfad171fd0d62cd4baab7b: Status 404 returned error can't find the container with id 2101a458fe9577f6e0850270113b1d478a04c65da0dfad171fd0d62cd4baab7b
Jul 16 21:49:52.468503 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.468326    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:44:51 +0000 UTC" deadline="2028-01-05 14:23:01.414893096 +0000 UTC"
Jul 16 21:49:52.468503 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.468495    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12904h33m8.946400938s"
Jul 16 21:49:52.556915 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.556827    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:52.557047 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:52.556919    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:49:52.564752 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.564731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jkm6h" event={"ID":"fb1570a9-59ea-4345-8408-5a192d1e24c6","Type":"ContainerStarted","Data":"edb972ea9782f277e6f567cc73d0a542b90f15360acfd4f3be4142244a46ed9a"}
Jul 16 21:49:52.565859 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.565839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"aa0354193979f69b800780009b05da1156031068f04a2e92d76d00f4bf5f4951"}
Jul 16 21:49:52.566748 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.566728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8jtl" event={"ID":"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5","Type":"ContainerStarted","Data":"5636a3bfbc79ace77b8ca2fe8ac1594f5ce03482c59db99203ddf7bbc3d262ad"}
Jul 16 21:49:52.567667 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.567645    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerStarted","Data":"cdb60e4e12d89e7237143854bee2a3ad9426b942aaf2ce9f48e000092ffdad69"}
Jul 16 21:49:52.568624 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.568606    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hh4jx" event={"ID":"e1160e2d-5f23-4b46-8369-316e2749a642","Type":"ContainerStarted","Data":"845534d1b2fb6d62b11ee8cd6449ca7665ecc916ff524459d64a852705f3ef8b"}
Jul 16 21:49:52.570335 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.570307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal" event={"ID":"6337947e1d545ee6d7cb3a765fe8fb4c","Type":"ContainerStarted","Data":"0d3bfcf6ad207d17c029e088d52eace005777b891d113d39e9ba9420c3c56d20"}
Jul 16 21:49:52.571773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.571751    2557 generic.go:356] "Generic (PLEG): container finished" podID="fa5496d3168ac96273becbca4a10ee28" containerID="1d260c40dceadb9896e7395c8867d2e6d0317bb24ff6a5a8fd47c84eb5b6730a" exitCode=0
Jul 16 21:49:52.571868 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.571829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" event={"ID":"fa5496d3168ac96273becbca4a10ee28","Type":"ContainerDied","Data":"1d260c40dceadb9896e7395c8867d2e6d0317bb24ff6a5a8fd47c84eb5b6730a"}
Jul 16 21:49:52.572924 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.572896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kqgqb" event={"ID":"35965ec5-69e1-42da-915d-2e123c7dc664","Type":"ContainerStarted","Data":"2101a458fe9577f6e0850270113b1d478a04c65da0dfad171fd0d62cd4baab7b"}
Jul 16 21:49:52.574022 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.573999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8bwn" event={"ID":"fba83155-ef2f-4233-a21b-155bb4abf982","Type":"ContainerStarted","Data":"44aac541011345bcfebcead449b6c4605013886e6fab837ad728ed5f5eafc2eb"}
Jul 16 21:49:52.575141 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.575114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6" event={"ID":"9e266e69-5ca7-42cc-8a40-773d457c2553","Type":"ContainerStarted","Data":"9c08ec045ebb75cecc0eea424bec070dcb40e7af9b96b205fdfcfa7cfd4ded2d"}
Jul 16 21:49:52.576065 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.576043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6z6pz" event={"ID":"52d062bf-25da-41c5-b1ae-0ecd872eaedf","Type":"ContainerStarted","Data":"70f35afc894ad2943e10135d8bb56ee455a5187a2aeadf829a0f57d8dd5edde6"}
Jul 16 21:49:52.584904 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:52.584861    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal" podStartSLOduration=2.584849744 podStartE2EDuration="2.584849744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:52.584536808 +0000 UTC m=+3.563886105" watchObservedRunningTime="2026-07-16 21:49:52.584849744 +0000 UTC m=+3.564199041"
Jul 16 21:49:53.040748 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.040711    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:53.040957 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.040782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:53.040957 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.040827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:53.041063 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.040959    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:53.041063 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.040978    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:53.041063 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.040990    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:53.041063 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.041049    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:49:55.041029777 +0000 UTC m=+6.020379054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:53.041489 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.041469    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:53.041592 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.041526    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:55.041511372 +0000 UTC m=+6.020860651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:53.041592 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.041590    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:53.041709 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.041627    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:55.041614181 +0000 UTC m=+6.020963460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:53.469113 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.469068    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:44:51 +0000 UTC" deadline="2027-12-17 16:22:37.388227437 +0000 UTC"
Jul 16 21:49:53.469113 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.469111    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12450h32m43.919120411s"
Jul 16 21:49:53.556961 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.556930    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:53.557117 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.557063    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:49:53.557488 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.557466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:53.557629 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:53.557566    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:49:53.601899 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:53.601861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" event={"ID":"fa5496d3168ac96273becbca4a10ee28","Type":"ContainerStarted","Data":"0f9b9a5543a17d9095d6965bfa2780167159bf19a1b63d06ba8b3c0e52e6c9bb"}
Jul 16 21:49:54.557699 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:54.557134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:54.557699 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:54.557269    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:49:55.059398 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:55.059359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:55.059582 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:55.059438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:55.059582 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:55.059475    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:55.059686 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.059616    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:55.059686 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.059634    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:55.059686 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.059646    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:55.059860 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.059700    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:49:59.059683034 +0000 UTC m=+10.039032317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:55.059986 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.059958    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:55.060040 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.060008    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:55.060040 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.060023    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:59.060005449 +0000 UTC m=+10.039354726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:55.060141 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.060068    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:59.060052188 +0000 UTC m=+10.039401469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:55.557678 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:55.557011    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:55.557678 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.557144    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:49:55.557678 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:55.557531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:55.557678 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:55.557637    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:49:56.557076 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:56.557040    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:56.557254 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:56.557224    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:49:57.556924 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:57.556888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:57.557348 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:57.557038    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:49:57.557542 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:57.557522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:57.557667 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:57.557644    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:49:58.556642 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:58.556604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:58.556861 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:58.556740    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:59.094866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:59.094930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:59.094964    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095048    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095088    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095095    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095106    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095120    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095131    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:07.095101267 +0000 UTC m=+18.074450549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095157    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:07.095138739 +0000 UTC m=+18.074488034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:59.095171 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.095176    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:50:07.0951665 +0000 UTC m=+18.074515778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:59.558323 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:59.557832    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:49:59.558323 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.557963    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:49:59.558323 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:49:59.558234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:49:59.558323 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:49:59.558293    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:00.556530 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:00.556486    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:00.557007 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:00.556614    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:01.557944 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:01.557918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:01.558429 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:01.558037    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:01.559584 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:01.559562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:01.559675 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:01.559660    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:02.556399 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:02.556362    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:02.556566 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:02.556508    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:03.556486 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:03.556453    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:03.556906 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:03.556581    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:03.556906 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:03.556607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:03.556906 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:03.556718    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:04.557027 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:04.556997    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:04.557485 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:04.557130    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:05.557197 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:05.557157    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:05.557669 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:05.557307    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:05.557669 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:05.557362    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:05.557669 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:05.557487    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:06.556439 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:06.556410    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:06.556655 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:06.556537    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:07.154041 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:07.153994    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:07.154052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:07.154106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154186    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154207    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154216    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154229    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154265    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:23.154246357 +0000 UTC m=+34.133595639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154284    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:50:23.154273957 +0000 UTC m=+34.133623239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154308    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:07.154557 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.154348    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:23.154332765 +0000 UTC m=+34.133682049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:07.556505 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:07.556423    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:07.556659 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.556552    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:07.556659 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:07.556638    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:07.556759 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:07.556742    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:08.556761 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:08.556726    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:08.557229 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:08.556852    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:09.558117 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:09.558084    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:09.558569 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:09.558173    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:09.558569 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:09.558195    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:09.558569 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:09.558305    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:10.556179 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:10.556145    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:10.556356 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:10.556261    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:11.557064 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.557037    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:11.557415 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:11.557168    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:11.557701 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.557680    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:11.557850 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:11.557803    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:11.646745 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.646082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8bwn" event={"ID":"fba83155-ef2f-4233-a21b-155bb4abf982","Type":"ContainerStarted","Data":"8162f599f74b8c54aca0df064910a82660bdf30f3fce92f8e46ef20aa42ed3d4"}
Jul 16 21:50:11.650427 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.649719    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8jtl" event={"ID":"502f8f6b-bf16-49dc-bbbf-ffba51fa81c5","Type":"ContainerStarted","Data":"df76781dacc79da69014f9bd066d0a8fa7890882fcbf073d84c49fb9693210b5"}
Jul 16 21:50:11.663793 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.663577    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q8bwn" podStartSLOduration=3.481475071 podStartE2EDuration="22.66355746s" totalImagesPullingTime="19.182082389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:11.663432929 +0000 UTC m=+22.642782228" watchObservedRunningTime="2026-07-16 21:50:11.66355746 +0000 UTC m=+22.642906757"
Jul 16 21:50:11.664381 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.664331    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" podStartSLOduration=21.664318512 podStartE2EDuration="21.664318512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:53.618357494 +0000 UTC m=+4.597706796" watchObservedRunningTime="2026-07-16 21:50:11.664318512 +0000 UTC m=+22.643667811"
Jul 16 21:50:11.684110 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:11.684058    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8jtl" podStartSLOduration=3.500952217 podStartE2EDuration="22.684040178s" totalImagesPullingTime="19.183087961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:11.683744577 +0000 UTC m=+22.663093877" watchObservedRunningTime="2026-07-16 21:50:11.684040178 +0000 UTC m=+22.663389477"
Jul 16 21:50:12.556729 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.556533    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:12.556927 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:12.556855    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:12.653567 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.653533    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6" event={"ID":"9e266e69-5ca7-42cc-8a40-773d457c2553","Type":"ContainerStarted","Data":"fd1c43e26173c3497ff3f0081e894ca830ac81cc39e7c6e13afaf990945aab87"}
Jul 16 21:50:12.654837 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.654791    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6z6pz" event={"ID":"52d062bf-25da-41c5-b1ae-0ecd872eaedf","Type":"ContainerStarted","Data":"1f89d3b84fea7cc447ec6b10c81352a4290fd4f3a6a3a8f256cf384f60b1a490"}
Jul 16 21:50:12.656128 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.656098    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jkm6h" event={"ID":"fb1570a9-59ea-4345-8408-5a192d1e24c6","Type":"ContainerStarted","Data":"33577fc543522cb6ac441a756f078bd335aef2776ab8d26647b4900cac203f98"}
Jul 16 21:50:12.658834 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.658786    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"57c7b039186ffdd88f13f97960ce2048ab4ae902e347711f3b52793486987fb9"}
Jul 16 21:50:12.658937 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.658868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"f5b1ae2868c803cd64d8064e966864f49cdb11a3b42f5ff9cc0c6d075b601f52"}
Jul 16 21:50:12.658937 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.658918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"0cea452e238bfce324e3b7eafe1fe0e071f0d23feb9253fbc90351f640ed18f4"}
Jul 16 21:50:12.659027 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.658951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"a5b9436ff234ae78f82b52d3782c1fddf89f005abf56117b0465bd3953e25011"}
Jul 16 21:50:12.659027 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.658977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"ab20f5540fa4826f5e83768b4e1fc69ccac67d0d4222048ddcb20b8b222689ef"}
Jul 16 21:50:12.659027 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.659018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"5f4e661ccf42ad9a1ac8ba6e8c42e3f118e091f710dd0995ea13c4a315f493ef"}
Jul 16 21:50:12.660201 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.660181    2557 generic.go:356] "Generic (PLEG): container finished" podID="92fc2188-8ee2-47ca-98bf-a477465b096d" containerID="eed44b299ee8449b8fa42146e4a04afb19f6a8850a7e5bafc741f44ab9d5ffbb" exitCode=0
Jul 16 21:50:12.660298 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.660242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerDied","Data":"eed44b299ee8449b8fa42146e4a04afb19f6a8850a7e5bafc741f44ab9d5ffbb"}
Jul 16 21:50:12.661592 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.661573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hh4jx" event={"ID":"e1160e2d-5f23-4b46-8369-316e2749a642","Type":"ContainerStarted","Data":"7164e1b1167dd9fd0e99a069d84fa29e2b733a8d8a4a8598c7e90b804dcd8371"}
Jul 16 21:50:12.671462 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.671428    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6z6pz" podStartSLOduration=4.486817183 podStartE2EDuration="23.671416832s" totalImagesPullingTime="19.184599649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:12.671098471 +0000 UTC m=+23.650447778" watchObservedRunningTime="2026-07-16 21:50:12.671416832 +0000 UTC m=+23.650766130"
Jul 16 21:50:12.719537 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.719495    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hh4jx" podStartSLOduration=4.499841866 podStartE2EDuration="23.719482058s" totalImagesPullingTime="19.219640192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:12.719415405 +0000 UTC m=+23.698764702" watchObservedRunningTime="2026-07-16 21:50:12.719482058 +0000 UTC m=+23.698831355"
Jul 16 21:50:12.738349 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:12.738319    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jkm6h" podStartSLOduration=4.515295816 podStartE2EDuration="23.738310006s" totalImagesPullingTime="19.22301419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:12.738198869 +0000 UTC m=+23.717548168" watchObservedRunningTime="2026-07-16 21:50:12.738310006 +0000 UTC m=+23.717659305"
Jul 16 21:50:13.045726 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.045697    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:50:13.488405 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.488287    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:50:13.045718895Z","UUID":"61dc1763-e07c-44b1-b105-f8ff8ccd9725","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:50:13.490368 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.490330    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:50:13.490368 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.490366    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:50:13.556560 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.556412    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:13.557101 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:13.556792    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:13.557101 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.556854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:13.557101 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:13.556939    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:13.580968 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.580946    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:50:13.581512 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.581496    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:50:13.667114 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.667082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kqgqb" event={"ID":"35965ec5-69e1-42da-915d-2e123c7dc664","Type":"ContainerStarted","Data":"9d15caf70a9ce019e8210eb3ae25c3dfae9b5e52267cbf1aa6b739e508b73239"}
Jul 16 21:50:13.669697 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.669664    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6" event={"ID":"9e266e69-5ca7-42cc-8a40-773d457c2553","Type":"ContainerStarted","Data":"542971d4effcc6ac9e5fee8bae780157bba22e17c1853b4c99c7e8a799964f61"}
Jul 16 21:50:13.683697 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:13.683651    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kqgqb" podStartSLOduration=5.504678226 podStartE2EDuration="24.683634138s" totalImagesPullingTime="19.178955912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:13.682896198 +0000 UTC m=+24.662245506" watchObservedRunningTime="2026-07-16 21:50:13.683634138 +0000 UTC m=+24.662983440"
Jul 16 21:50:14.556824 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:14.556796    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:14.556944 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:14.556925    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:14.677611 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:14.676870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6" event={"ID":"9e266e69-5ca7-42cc-8a40-773d457c2553","Type":"ContainerStarted","Data":"1f63ac3884e848cebc74ca3f501c330685960703e5e867fbb1b9187a371375dc"}
Jul 16 21:50:14.682003 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:14.681981    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"1e93adc22a9fb5deb5b16cfe272075b29f19e082e58a9e6c75e9d51de4b79f22"}
Jul 16 21:50:14.682091 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:14.682062    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 16 21:50:15.556609 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:15.556418    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:15.556893 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:15.556460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:15.556893 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:15.556707    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:15.556893 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:15.556805    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:15.702920 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:15.702871    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctmn6" podStartSLOduration=4.802693392 podStartE2EDuration="26.702852263s" totalImagesPullingTime="21.900158871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:15.702809044 +0000 UTC m=+26.682158355" watchObservedRunningTime="2026-07-16 21:50:15.702852263 +0000 UTC m=+26.682201556"
Jul 16 21:50:16.557100 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:16.557069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:16.557269 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:16.557162    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:17.557205 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.557174    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:17.558076 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.557176    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:17.558076 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:17.557335    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:17.558076 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:17.557376    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:17.691608 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.691581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" event={"ID":"29e8bd1d-6583-4173-9236-ab02fba76157","Type":"ContainerStarted","Data":"6fb2961ac794368bf24ec54b893aa13fe7258977349aa2647782018fe7129665"}
Jul 16 21:50:17.691847 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.691823    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:50:17.693598 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.693573    2557 generic.go:356] "Generic (PLEG): container finished" podID="92fc2188-8ee2-47ca-98bf-a477465b096d" containerID="b8b235aeb06d66f8701d86a47daeba7b7bd196b8f44e5442758822ebe088dc90" exitCode=0
Jul 16 21:50:17.693702 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.693615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerDied","Data":"b8b235aeb06d66f8701d86a47daeba7b7bd196b8f44e5442758822ebe088dc90"}
Jul 16 21:50:17.707376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.707236    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:50:17.720124 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:17.720089    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2" podStartSLOduration=9.354448388 podStartE2EDuration="28.720077598s" totalImagesPullingTime="19.36562921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:17.718683927 +0000 UTC m=+28.698033224" watchObservedRunningTime="2026-07-16 21:50:17.720077598 +0000 UTC m=+28.699426895"
Jul 16 21:50:18.557184 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.557156    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:18.557431 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:18.557259    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:18.696244 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.696113    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:50:18.696244 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.696193    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:50:18.714308 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.714276    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:50:18.987712 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.987685    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cpdkm"]
Jul 16 21:50:18.987859 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.987848    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:18.987973 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:18.987951    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:18.988391 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.988370    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dpnxv"]
Jul 16 21:50:18.988501 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.988488    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:18.988588 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:18.988571    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:18.989011 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.988993    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sr4st"]
Jul 16 21:50:18.989099 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:18.989063    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:18.989141 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:18.989116    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:19.033853 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:19.033832    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:50:19.033951 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:19.033937    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 16 21:50:19.034334 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:19.034321    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q8bwn"
Jul 16 21:50:19.699980 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:19.699952    2557 generic.go:356] "Generic (PLEG): container finished" podID="92fc2188-8ee2-47ca-98bf-a477465b096d" containerID="7af8e24f0494aa7e3997a3f340c82c7beec78f4b03dcf5aa77315016b8251895" exitCode=0
Jul 16 21:50:19.700432 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:19.700041    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerDied","Data":"7af8e24f0494aa7e3997a3f340c82c7beec78f4b03dcf5aa77315016b8251895"}
Jul 16 21:50:20.556502 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:20.556467    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:20.556674 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:20.556558    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:20.556674 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:20.556661    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:20.556777 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:20.556564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:20.556777 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:20.556560    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:20.556898 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:20.556782    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:21.706302 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:21.706272    2557 generic.go:356] "Generic (PLEG): container finished" podID="92fc2188-8ee2-47ca-98bf-a477465b096d" containerID="68deb7be5c52e0dfe1e4bb4bdf3ff90d72e3f5ea696695f240b3b6f16ee8eb72" exitCode=0
Jul 16 21:50:21.706697 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:21.706326    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerDied","Data":"68deb7be5c52e0dfe1e4bb4bdf3ff90d72e3f5ea696695f240b3b6f16ee8eb72"}
Jul 16 21:50:22.557452 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:22.557214    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:22.557627 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:22.557276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:22.557627 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:22.557556    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:22.557627 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:22.557277    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:22.557792 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:22.557694    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:22.557792 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:22.557773    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:23.172827 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:23.172775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:23.172863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:23.172886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.172928    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.172994    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.173008    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.173021    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.173000    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret podName:4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:55.172979915 +0000 UTC m=+66.152329204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret") pod "global-pull-secret-syncer-cpdkm" (UID: "4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.173032    2557 projected.go:196] Error preparing data for projected volume kube-api-access-ktmmh for pod openshift-network-diagnostics/network-check-target-sr4st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.173056    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs podName:24fdc43a-8826-46d0-8c18-ace05937ce99 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:55.173039191 +0000 UTC m=+66.152388471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs") pod "network-metrics-daemon-dpnxv" (UID: "24fdc43a-8826-46d0-8c18-ace05937ce99") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:23.173298 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:23.173072    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh podName:8c1eb6f4-cde0-4dad-a483-e8e265add7ce nodeName:}" failed. No retries permitted until 2026-07-16 21:50:55.173062909 +0000 UTC m=+66.152412189 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktmmh" (UniqueName: "kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh") pod "network-check-target-sr4st" (UID: "8c1eb6f4-cde0-4dad-a483-e8e265add7ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:24.556761 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:24.556720    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:24.557206 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:24.556726    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:24.557206 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:24.556859    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpnxv" podUID="24fdc43a-8826-46d0-8c18-ace05937ce99"
Jul 16 21:50:24.557206 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:24.556960    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpdkm" podUID="4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2"
Jul 16 21:50:24.557206 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:24.556734    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:24.557206 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:24.557054    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4st" podUID="8c1eb6f4-cde0-4dad-a483-e8e265add7ce"
Jul 16 21:50:25.373990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.373908    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeReady"
Jul 16 21:50:25.374141 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.374066    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:50:25.424362 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.424280    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"]
Jul 16 21:50:25.426944 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.426913    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.429988 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.429964    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:50:25.430975 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.430957    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:50:25.431097 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.431072    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-47gk2\""
Jul 16 21:50:25.435114 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.434879    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:50:25.440839 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.440795    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:50:25.443966 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.443943    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"]
Jul 16 21:50:25.468318 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.467513    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x9px6"]
Jul 16 21:50:25.469473 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.469447    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8rcrk"]
Jul 16 21:50:25.469606 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.469580    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.471196 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.471179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.480791 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.480771    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g56tn\""
Jul 16 21:50:25.480791 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.480788    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:50:25.482179 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.482158    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:50:25.482307 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.482278    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:50:25.482396 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.482159    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vqlsj\""
Jul 16 21:50:25.482527 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.482510    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:50:25.482664 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.482651    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:50:25.483335 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.483316    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rcrk"]
Jul 16 21:50:25.490570 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.490542    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9px6"]
Jul 16 21:50:25.492808 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.492776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-image-registry-private-configuration\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.492808 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.492830    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-registry-certificates\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.492997 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.492857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxm25\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-kube-api-access-wxm25\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.492997 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.492893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a79de3dd-836d-4018-9482-974abea12707-ca-trust-extracted\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.492997 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.492978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-trusted-ca\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.493193 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.493029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.493193 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.493053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-installation-pull-secrets\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.493193 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.493082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-bound-sa-token\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593444 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f65c0253-132a-4143-9015-1dd343dd797b-config-volume\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.593444 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593447    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97aa7d52-d173-4110-90be-35f610bdda8f-cert\") pod \"ingress-canary-8rcrk\" (UID: \"97aa7d52-d173-4110-90be-35f610bdda8f\") " pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wwx\" (UniqueName: \"kubernetes.io/projected/97aa7d52-d173-4110-90be-35f610bdda8f-kube-api-access-h6wwx\") pod \"ingress-canary-8rcrk\" (UID: \"97aa7d52-d173-4110-90be-35f610bdda8f\") " pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-image-registry-private-configuration\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-registry-certificates\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxm25\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-kube-api-access-wxm25\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a79de3dd-836d-4018-9482-974abea12707-ca-trust-extracted\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzs26\" (UniqueName: \"kubernetes.io/projected/f65c0253-132a-4143-9015-1dd343dd797b-kube-api-access-fzs26\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-trusted-ca\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593789    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593829    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65c0253-132a-4143-9015-1dd343dd797b-tmp-dir\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593928    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-installation-pull-secrets\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.593981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.593960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-bound-sa-token\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.594504 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:25.594012    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:25.594504 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:25.594026    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd5bf47b9-b6s8g: secret "image-registry-tls" not found
Jul 16 21:50:25.594504 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:25.594076    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls podName:a79de3dd-836d-4018-9482-974abea12707 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:26.094059327 +0000 UTC m=+37.073408604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls") pod "image-registry-5cd5bf47b9-b6s8g" (UID: "a79de3dd-836d-4018-9482-974abea12707") : secret "image-registry-tls" not found
Jul 16 21:50:25.594504 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.594230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-registry-certificates\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.594654 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.594633    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a79de3dd-836d-4018-9482-974abea12707-ca-trust-extracted\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.594707 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.594688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-trusted-ca\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.598420 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.598396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-image-registry-private-configuration\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.598535 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.598440    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-installation-pull-secrets\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.604393 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.604372    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxm25\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-kube-api-access-wxm25\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.604486 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.604373    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-bound-sa-token\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:25.695007 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.694981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzs26\" (UniqueName: \"kubernetes.io/projected/f65c0253-132a-4143-9015-1dd343dd797b-kube-api-access-fzs26\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.695129 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.695129 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65c0253-132a-4143-9015-1dd343dd797b-tmp-dir\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.695129 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f65c0253-132a-4143-9015-1dd343dd797b-config-volume\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.695280 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:25.695198    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:25.695280 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:25.695270    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls podName:f65c0253-132a-4143-9015-1dd343dd797b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:26.195249486 +0000 UTC m=+37.174598784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls") pod "dns-default-x9px6" (UID: "f65c0253-132a-4143-9015-1dd343dd797b") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:25.695378 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97aa7d52-d173-4110-90be-35f610bdda8f-cert\") pod \"ingress-canary-8rcrk\" (UID: \"97aa7d52-d173-4110-90be-35f610bdda8f\") " pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.695378 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wwx\" (UniqueName: \"kubernetes.io/projected/97aa7d52-d173-4110-90be-35f610bdda8f-kube-api-access-h6wwx\") pod \"ingress-canary-8rcrk\" (UID: \"97aa7d52-d173-4110-90be-35f610bdda8f\") " pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.695482 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695445    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65c0253-132a-4143-9015-1dd343dd797b-tmp-dir\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.695703 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.695683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f65c0253-132a-4143-9015-1dd343dd797b-config-volume\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.697773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.697750    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97aa7d52-d173-4110-90be-35f610bdda8f-cert\") pod \"ingress-canary-8rcrk\" (UID: \"97aa7d52-d173-4110-90be-35f610bdda8f\") " pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.707804 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.707779    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wwx\" (UniqueName: \"kubernetes.io/projected/97aa7d52-d173-4110-90be-35f610bdda8f-kube-api-access-h6wwx\") pod \"ingress-canary-8rcrk\" (UID: \"97aa7d52-d173-4110-90be-35f610bdda8f\") " pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.707964 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.707859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzs26\" (UniqueName: \"kubernetes.io/projected/f65c0253-132a-4143-9015-1dd343dd797b-kube-api-access-fzs26\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:25.787484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.787458    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rcrk"
Jul 16 21:50:25.924648 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:25.924476    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rcrk"]
Jul 16 21:50:25.929105 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:25.929080    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aa7d52_d173_4110_90be_35f610bdda8f.slice/crio-0043e218b8911aa02139a0640641d0599f2809b2861a9396d420179d3957bdb1 WatchSource:0}: Error finding container 0043e218b8911aa02139a0640641d0599f2809b2861a9396d420179d3957bdb1: Status 404 returned error can't find the container with id 0043e218b8911aa02139a0640641d0599f2809b2861a9396d420179d3957bdb1
Jul 16 21:50:26.098844 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.098794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:26.099021 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:26.098959    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:26.099021 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:26.098981    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd5bf47b9-b6s8g: secret "image-registry-tls" not found
Jul 16 21:50:26.099098 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:26.099046    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls podName:a79de3dd-836d-4018-9482-974abea12707 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:27.099031015 +0000 UTC m=+38.078380291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls") pod "image-registry-5cd5bf47b9-b6s8g" (UID: "a79de3dd-836d-4018-9482-974abea12707") : secret "image-registry-tls" not found
Jul 16 21:50:26.199801 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.199714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:26.199966 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:26.199913    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:26.200043 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:26.199989    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls podName:f65c0253-132a-4143-9015-1dd343dd797b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:27.199969917 +0000 UTC m=+38.179319194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls") pod "dns-default-x9px6" (UID: "f65c0253-132a-4143-9015-1dd343dd797b") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:26.557119 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.557043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:26.557293 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.557043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:26.557293 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.557043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:26.559989 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.559969    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn8m8\""
Jul 16 21:50:26.560138 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.560039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:50:26.560138 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.560084    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:50:26.560138 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.560133    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:50:26.560773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.560753    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qpnd8\""
Jul 16 21:50:26.560907 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.560885    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:50:26.720535 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.720504    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rcrk" event={"ID":"97aa7d52-d173-4110-90be-35f610bdda8f","Type":"ContainerStarted","Data":"0043e218b8911aa02139a0640641d0599f2809b2861a9396d420179d3957bdb1"}
Jul 16 21:50:26.723636 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.723607    2557 generic.go:356] "Generic (PLEG): container finished" podID="92fc2188-8ee2-47ca-98bf-a477465b096d" containerID="8d95f86cd8f035ac35bb723f9510884ea64cf253fc45fd0dc8fa8b40fe0b3577" exitCode=0
Jul 16 21:50:26.723769 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:26.723653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerDied","Data":"8d95f86cd8f035ac35bb723f9510884ea64cf253fc45fd0dc8fa8b40fe0b3577"}
Jul 16 21:50:27.106704 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:27.106614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:27.106879 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:27.106796    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:27.106879 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:27.106830    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd5bf47b9-b6s8g: secret "image-registry-tls" not found
Jul 16 21:50:27.107003 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:27.106898    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls podName:a79de3dd-836d-4018-9482-974abea12707 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:29.106877382 +0000 UTC m=+40.086226659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls") pod "image-registry-5cd5bf47b9-b6s8g" (UID: "a79de3dd-836d-4018-9482-974abea12707") : secret "image-registry-tls" not found
Jul 16 21:50:27.207851 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:27.207803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:27.208011 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:27.207932    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:27.208011 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:27.208005    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls podName:f65c0253-132a-4143-9015-1dd343dd797b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:29.207978815 +0000 UTC m=+40.187328094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls") pod "dns-default-x9px6" (UID: "f65c0253-132a-4143-9015-1dd343dd797b") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:27.730150 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:27.730112    2557 generic.go:356] "Generic (PLEG): container finished" podID="92fc2188-8ee2-47ca-98bf-a477465b096d" containerID="5fd214737c69954358624a84995d67ef96c3d27fa24ad58b339dac39acfc6163" exitCode=0
Jul 16 21:50:27.730558 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:27.730174    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerDied","Data":"5fd214737c69954358624a84995d67ef96c3d27fa24ad58b339dac39acfc6163"}
Jul 16 21:50:28.737015 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:28.736833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m95lw" event={"ID":"92fc2188-8ee2-47ca-98bf-a477465b096d","Type":"ContainerStarted","Data":"fe7025592e69cb42129f188a06125c67b997447000a7872427ab8b721f1044b8"}
Jul 16 21:50:28.738414 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:28.738388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rcrk" event={"ID":"97aa7d52-d173-4110-90be-35f610bdda8f","Type":"ContainerStarted","Data":"d45bc92cb7414f119b3d61bb1c4e88f5a82202c535b1afeb5da249aee459619a"}
Jul 16 21:50:28.761416 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:28.761376    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m95lw" podStartSLOduration=10.261911981 podStartE2EDuration="39.761362495s" totalImagesPullingTime="29.499450514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:28.759998649 +0000 UTC m=+39.739347948" watchObservedRunningTime="2026-07-16 21:50:28.761362495 +0000 UTC m=+39.740711793"
Jul 16 21:50:28.776712 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:28.776674    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8rcrk" podStartSLOduration=1.610323275 podStartE2EDuration="3.776659902s" totalImagesPullingTime="2.166336627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:28.776346496 +0000 UTC m=+39.755695796" watchObservedRunningTime="2026-07-16 21:50:28.776659902 +0000 UTC m=+39.756009200"
Jul 16 21:50:29.123130 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.123043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:29.123257 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.123197    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:29.123257 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.123214    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd5bf47b9-b6s8g: secret "image-registry-tls" not found
Jul 16 21:50:29.123334 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.123271    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls podName:a79de3dd-836d-4018-9482-974abea12707 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.123257434 +0000 UTC m=+44.102606711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls") pod "image-registry-5cd5bf47b9-b6s8g" (UID: "a79de3dd-836d-4018-9482-974abea12707") : secret "image-registry-tls" not found
Jul 16 21:50:29.224131 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.224100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:29.224274 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.224227    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:29.224315 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.224278    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls podName:f65c0253-132a-4143-9015-1dd343dd797b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.224265521 +0000 UTC m=+44.203614798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls") pod "dns-default-x9px6" (UID: "f65c0253-132a-4143-9015-1dd343dd797b") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:29.597597 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.597560    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"]
Jul 16 21:50:29.618981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.618954    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"]
Jul 16 21:50:29.619128 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.619054    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.625557 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.625438    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.625557 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.625461    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dtn4x\""
Jul 16 21:50:29.625753 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.625732    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 21:50:29.625866 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.625783    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 21:50:29.625866 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.625808    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.700827 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.700783    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"]
Jul 16 21:50:29.722510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.722484    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-t4lc2"]
Jul 16 21:50:29.722665 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.722645    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"
Jul 16 21:50:29.725302 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.725283    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.725726 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.725711    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vpccq\""
Jul 16 21:50:29.726626 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.726598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pd2\" (UniqueName: \"kubernetes.io/projected/37dae182-a8ff-48b8-898e-155bda8bf0c5-kube-api-access-l8pd2\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.726736 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.726699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/37dae182-a8ff-48b8-898e-155bda8bf0c5-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.726846 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.726802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.726976 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.726961    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.746663 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.746641    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"]
Jul 16 21:50:29.747031 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.746678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-t4lc2"]
Jul 16 21:50:29.747031 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.746699    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-559484955d-q46dq"]
Jul 16 21:50:29.747031 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.746745    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.750603 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.750587    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 16 21:50:29.750985 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.750965    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l8xnn\""
Jul 16 21:50:29.751077 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.751009    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.751264 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.751232    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 16 21:50:29.753867 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.753850    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.756944 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.756925    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 16 21:50:29.770583 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.770564    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-559484955d-q46dq"]
Jul 16 21:50:29.770653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.770617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:29.773808 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.773790    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 21:50:29.773893 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.773805    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 21:50:29.774132 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.774116    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.774404 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.774388    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 21:50:29.774404 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.774398    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.774528 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.774396    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 21:50:29.775770 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.775755    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xtb2b\""
Jul 16 21:50:29.808743 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.808710    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"]
Jul 16 21:50:29.827233 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827201    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/37dae182-a8ff-48b8-898e-155bda8bf0c5-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.827410 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827242    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.827410 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.827381    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:29.827410 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3afdced7-a64f-4cac-858a-362b32874bd5-tmp\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.827518 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afdced7-a64f-4cac-858a-362b32874bd5-service-ca-bundle\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.827518 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:29.827467    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls podName:37dae182-a8ff-48b8-898e-155bda8bf0c5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:30.327423049 +0000 UTC m=+41.306772347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-nkrfq" (UID: "37dae182-a8ff-48b8-898e-155bda8bf0c5") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:29.827751 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv79f\" (UniqueName: \"kubernetes.io/projected/a3d0824f-6316-40f5-8268-d138a1d9aa61-kube-api-access-rv79f\") pod \"volume-data-source-validator-78cf4c6f6-scjs9\" (UID: \"a3d0824f-6316-40f5-8268-d138a1d9aa61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"
Jul 16 21:50:29.827867 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3afdced7-a64f-4cac-858a-362b32874bd5-snapshots\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.828510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afdced7-a64f-4cac-858a-362b32874bd5-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.828510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.827926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pd2\" (UniqueName: \"kubernetes.io/projected/37dae182-a8ff-48b8-898e-155bda8bf0c5-kube-api-access-l8pd2\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.828510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.828011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3afdced7-a64f-4cac-858a-362b32874bd5-serving-cert\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.828510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.828048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99d2x\" (UniqueName: \"kubernetes.io/projected/3afdced7-a64f-4cac-858a-362b32874bd5-kube-api-access-99d2x\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.828510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.828236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/37dae182-a8ff-48b8-898e-155bda8bf0c5-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.834534 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.834509    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-t7tsg"]
Jul 16 21:50:29.834640 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.834623    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:29.837465 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.837440    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6jz9n\""
Jul 16 21:50:29.837561 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.837447    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 16 21:50:29.837616 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.837580    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 16 21:50:29.838340 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.838314    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.838340 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.838337    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.849782 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.849729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pd2\" (UniqueName: \"kubernetes.io/projected/37dae182-a8ff-48b8-898e-155bda8bf0c5-kube-api-access-l8pd2\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:29.854754 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.854737    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"]
Jul 16 21:50:29.854898 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.854873    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:29.857776 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.857759    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 16 21:50:29.857776 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.857769    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.858212 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.858199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-25kp5\""
Jul 16 21:50:29.858276 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.858201    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.858948 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.858932    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 16 21:50:29.867695 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.867671    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 16 21:50:29.876876 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.876854    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"]
Jul 16 21:50:29.876975 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.876960    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:29.880264 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.880243    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 16 21:50:29.880445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.880427    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.883114 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.883096    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lgwlg\""
Jul 16 21:50:29.883299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.883276    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.883407 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.883313    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 16 21:50:29.902977 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.902954    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"]
Jul 16 21:50:29.903085 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.902994    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-t7tsg"]
Jul 16 21:50:29.903085 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.903014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"]
Jul 16 21:50:29.903085 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.903031    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"]
Jul 16 21:50:29.903085 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.902993    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:29.905405 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.905375    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 21:50:29.905511 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.905422    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xf2sl\""
Jul 16 21:50:29.905511 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.905375    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:29.905511 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.905433    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:29.928953 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.928926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmgp\" (UniqueName: \"kubernetes.io/projected/5af2916e-742e-4253-a404-425b475b1863-kube-api-access-prmgp\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:29.929073 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.928968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3afdced7-a64f-4cac-858a-362b32874bd5-serving-cert\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929073 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.928987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:29.929073 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:29.929073 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99d2x\" (UniqueName: \"kubernetes.io/projected/3afdced7-a64f-4cac-858a-362b32874bd5-kube-api-access-99d2x\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929189 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5af2916e-742e-4253-a404-425b475b1863-serving-cert\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:29.929189 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3afdced7-a64f-4cac-858a-362b32874bd5-tmp\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929251 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929188    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afdced7-a64f-4cac-858a-362b32874bd5-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929251 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929216    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hdc\" (UniqueName: \"kubernetes.io/projected/ec73e762-8ce7-42b8-b8b4-bad1d863c480-kube-api-access-h2hdc\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:29.929369 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af2916e-742e-4253-a404-425b475b1863-trusted-ca\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:29.929473 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929386    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-stats-auth\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:29.929473 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-serving-cert\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:29.929473 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afdced7-a64f-4cac-858a-362b32874bd5-service-ca-bundle\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929492    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3afdced7-a64f-4cac-858a-362b32874bd5-tmp\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929497    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgzl\" (UniqueName: \"kubernetes.io/projected/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-kube-api-access-4lgzl\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:29.929629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-default-certificate\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:29.929629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929568    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv79f\" (UniqueName: \"kubernetes.io/projected/a3d0824f-6316-40f5-8268-d138a1d9aa61-kube-api-access-rv79f\") pod \"volume-data-source-validator-78cf4c6f6-scjs9\" (UID: \"a3d0824f-6316-40f5-8268-d138a1d9aa61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"
Jul 16 21:50:29.929629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5af2916e-742e-4253-a404-425b475b1863-config\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:29.929629 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3afdced7-a64f-4cac-858a-362b32874bd5-snapshots\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.929946 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-config\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:29.930005 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.929976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afdced7-a64f-4cac-858a-362b32874bd5-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.930058 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.930039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3afdced7-a64f-4cac-858a-362b32874bd5-service-ca-bundle\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.930200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.930181    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3afdced7-a64f-4cac-858a-362b32874bd5-snapshots\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.931289 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.931265    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3afdced7-a64f-4cac-858a-362b32874bd5-serving-cert\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:29.938781 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.938751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv79f\" (UniqueName: \"kubernetes.io/projected/a3d0824f-6316-40f5-8268-d138a1d9aa61-kube-api-access-rv79f\") pod \"volume-data-source-validator-78cf4c6f6-scjs9\" (UID: \"a3d0824f-6316-40f5-8268-d138a1d9aa61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"
Jul 16 21:50:29.940459 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:29.940437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99d2x\" (UniqueName: \"kubernetes.io/projected/3afdced7-a64f-4cac-858a-362b32874bd5-kube-api-access-99d2x\") pod \"insights-operator-555d9d7885-t4lc2\" (UID: \"3afdced7-a64f-4cac-858a-362b32874bd5\") " pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:30.030575 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030544    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5af2916e-742e-4253-a404-425b475b1863-config\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.030704 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h44b\" (UniqueName: \"kubernetes.io/projected/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-kube-api-access-8h44b\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.030704 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-config\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.030704 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.030875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030761    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmgp\" (UniqueName: \"kubernetes.io/projected/5af2916e-742e-4253-a404-425b475b1863-kube-api-access-prmgp\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.030875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.030972 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.030972 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.030972 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.030955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5af2916e-742e-4253-a404-425b475b1863-serving-cert\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.031104 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hdc\" (UniqueName: \"kubernetes.io/projected/ec73e762-8ce7-42b8-b8b4-bad1d863c480-kube-api-access-h2hdc\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.031104 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.031027    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:50:30.031104 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7k8\" (UniqueName: \"kubernetes.io/projected/7b82138a-8001-4ce5-80d6-6843f4ced688-kube-api-access-dc7k8\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:30.031104 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.031054    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:30.53103509 +0000 UTC m=+41.510384385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : configmap references non-existent config key: service-ca.crt
Jul 16 21:50:30.031296 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.031171    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:30.531157195 +0000 UTC m=+41.510506474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : secret "router-metrics-certs-default" not found
Jul 16 21:50:30.031296 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af2916e-742e-4253-a404-425b475b1863-trusted-ca\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.031296 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:30.031296 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031260    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-config\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.031474 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031308    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-stats-auth\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.031474 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-serving-cert\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.031474 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031388    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgzl\" (UniqueName: \"kubernetes.io/projected/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-kube-api-access-4lgzl\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.031474 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5af2916e-742e-4253-a404-425b475b1863-config\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.031474 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-default-certificate\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.031709 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.031554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"
Jul 16 21:50:30.032474 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.032447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af2916e-742e-4253-a404-425b475b1863-trusted-ca\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.033638 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.033613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5af2916e-742e-4253-a404-425b475b1863-serving-cert\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.033950 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.033930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-serving-cert\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.034019 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.033938    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-default-certificate\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.034205 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.034189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-stats-auth\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.039867 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.039842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgzl\" (UniqueName: \"kubernetes.io/projected/fb11657a-5fd6-4f2d-8c12-fefc715c6bab-kube-api-access-4lgzl\") pod \"service-ca-operator-5bcbc94c78-p2tm6\" (UID: \"fb11657a-5fd6-4f2d-8c12-fefc715c6bab\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.040485 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.040465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hdc\" (UniqueName: \"kubernetes.io/projected/ec73e762-8ce7-42b8-b8b4-bad1d863c480-kube-api-access-h2hdc\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.041400 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.041379    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmgp\" (UniqueName: \"kubernetes.io/projected/5af2916e-742e-4253-a404-425b475b1863-kube-api-access-prmgp\") pod \"console-operator-565b7b569d-t7tsg\" (UID: \"5af2916e-742e-4253-a404-425b475b1863\") " pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.055930 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.055911    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-t4lc2"
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.131962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h44b\" (UniqueName: \"kubernetes.io/projected/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-kube-api-access-8h44b\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.132017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.132088    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.132178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7k8\" (UniqueName: \"kubernetes.io/projected/7b82138a-8001-4ce5-80d6-6843f4ced688-kube-api-access-dc7k8\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.132211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.132403    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:50:30.132868 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.132470    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls podName:7b82138a-8001-4ce5-80d6-6843f4ced688 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:30.63245106 +0000 UTC m=+41.611800361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ft7w4" (UID: "7b82138a-8001-4ce5-80d6-6843f4ced688") : secret "samples-operator-tls" not found
Jul 16 21:50:30.135056 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.134961    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.135877 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.135851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.144367 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.144338    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"
Jul 16 21:50:30.159500 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.159475    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7k8\" (UniqueName: \"kubernetes.io/projected/7b82138a-8001-4ce5-80d6-6843f4ced688-kube-api-access-dc7k8\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:30.159707 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.159683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h44b\" (UniqueName: \"kubernetes.io/projected/8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c-kube-api-access-8h44b\") pod \"kube-storage-version-migrator-operator-7658d9b749-vtvvf\" (UID: \"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.163468 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.163430    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:30.185956 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.185843    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"
Jul 16 21:50:30.192041 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.190604    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9"]
Jul 16 21:50:30.192041 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:30.191000    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d0824f_6316_40f5_8268_d138a1d9aa61.slice/crio-19f939e617bf5d65186e1f3a5f9eda9f48a07eadc3540c6685e4e516862d1645 WatchSource:0}: Error finding container 19f939e617bf5d65186e1f3a5f9eda9f48a07eadc3540c6685e4e516862d1645: Status 404 returned error can't find the container with id 19f939e617bf5d65186e1f3a5f9eda9f48a07eadc3540c6685e4e516862d1645
Jul 16 21:50:30.207184 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.207154    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-t4lc2"]
Jul 16 21:50:30.211795 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:30.211770    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afdced7_a64f_4cac_858a_362b32874bd5.slice/crio-d9095589664d0caa025e26c58755d7028f754f37d5d8246b22c240db0dc309f4 WatchSource:0}: Error finding container d9095589664d0caa025e26c58755d7028f754f37d5d8246b22c240db0dc309f4: Status 404 returned error can't find the container with id d9095589664d0caa025e26c58755d7028f754f37d5d8246b22c240db0dc309f4
Jul 16 21:50:30.323534 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.323502    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6"]
Jul 16 21:50:30.326469 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:30.326439    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb11657a_5fd6_4f2d_8c12_fefc715c6bab.slice/crio-c90bc447df68ec875ae0986a149d924e771ca5ecf66be1d3441bf37aa138715a WatchSource:0}: Error finding container c90bc447df68ec875ae0986a149d924e771ca5ecf66be1d3441bf37aa138715a: Status 404 returned error can't find the container with id c90bc447df68ec875ae0986a149d924e771ca5ecf66be1d3441bf37aa138715a
Jul 16 21:50:30.334512 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.334340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:30.334635 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.334457    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:30.334731 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.334682    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls podName:37dae182-a8ff-48b8-898e-155bda8bf0c5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:31.334661316 +0000 UTC m=+42.314010614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-nkrfq" (UID: "37dae182-a8ff-48b8-898e-155bda8bf0c5") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:30.346875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.346849    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-t7tsg"]
Jul 16 21:50:30.350222 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:30.350199    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af2916e_742e_4253_a404_425b475b1863.slice/crio-a7b5b985697feab4c123ce60ca90ddab7235e7dee47d353264f7c924f2bbcbc0 WatchSource:0}: Error finding container a7b5b985697feab4c123ce60ca90ddab7235e7dee47d353264f7c924f2bbcbc0: Status 404 returned error can't find the container with id a7b5b985697feab4c123ce60ca90ddab7235e7dee47d353264f7c924f2bbcbc0
Jul 16 21:50:30.381484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.381410    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf"]
Jul 16 21:50:30.384564 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:30.384542    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff8e0a6_ed66_4e44_9aa9_ff1c2271a73c.slice/crio-8b18d14f0d9a1cf1586a99d60dfb42636a8620db295172687e8528715ae51a99 WatchSource:0}: Error finding container 8b18d14f0d9a1cf1586a99d60dfb42636a8620db295172687e8528715ae51a99: Status 404 returned error can't find the container with id 8b18d14f0d9a1cf1586a99d60dfb42636a8620db295172687e8528715ae51a99
Jul 16 21:50:30.536945 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.536909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.536945 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.536949    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:30.537146 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.537061    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:50:30.537146 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.537121    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:31.537108209 +0000 UTC m=+42.516457485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : configmap references non-existent config key: service-ca.crt
Jul 16 21:50:30.537146 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.537136    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:31.537129744 +0000 UTC m=+42.516479020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : secret "router-metrics-certs-default" not found
Jul 16 21:50:30.638487 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.638404    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:30.638611 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.638556    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:50:30.638645 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:30.638618    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls podName:7b82138a-8001-4ce5-80d6-6843f4ced688 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:31.638603006 +0000 UTC m=+42.617952303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ft7w4" (UID: "7b82138a-8001-4ce5-80d6-6843f4ced688") : secret "samples-operator-tls" not found
Jul 16 21:50:30.744773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.744738    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf" event={"ID":"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c","Type":"ContainerStarted","Data":"8b18d14f0d9a1cf1586a99d60dfb42636a8620db295172687e8528715ae51a99"}
Jul 16 21:50:30.745961 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.745936    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6" event={"ID":"fb11657a-5fd6-4f2d-8c12-fefc715c6bab","Type":"ContainerStarted","Data":"c90bc447df68ec875ae0986a149d924e771ca5ecf66be1d3441bf37aa138715a"}
Jul 16 21:50:30.747156 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.747138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-t4lc2" event={"ID":"3afdced7-a64f-4cac-858a-362b32874bd5","Type":"ContainerStarted","Data":"d9095589664d0caa025e26c58755d7028f754f37d5d8246b22c240db0dc309f4"}
Jul 16 21:50:30.748335 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.748306    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9" event={"ID":"a3d0824f-6316-40f5-8268-d138a1d9aa61","Type":"ContainerStarted","Data":"19f939e617bf5d65186e1f3a5f9eda9f48a07eadc3540c6685e4e516862d1645"}
Jul 16 21:50:30.749293 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:30.749274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" event={"ID":"5af2916e-742e-4253-a404-425b475b1863","Type":"ContainerStarted","Data":"a7b5b985697feab4c123ce60ca90ddab7235e7dee47d353264f7c924f2bbcbc0"}
Jul 16 21:50:31.346795 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:31.346070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:31.346795 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.346393    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:31.346795 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.346455    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls podName:37dae182-a8ff-48b8-898e-155bda8bf0c5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.346436725 +0000 UTC m=+44.325786016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-nkrfq" (UID: "37dae182-a8ff-48b8-898e-155bda8bf0c5") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:31.548075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:31.547913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:31.548271 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.548080    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.548058172 +0000 UTC m=+44.527407454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : configmap references non-existent config key: service-ca.crt
Jul 16 21:50:31.548271 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:31.548176    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:31.548394 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.548293    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:50:31.548394 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.548331    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.548319839 +0000 UTC m=+44.527669119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : secret "router-metrics-certs-default" not found
Jul 16 21:50:31.648963 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:31.648912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:31.649207 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.649081    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:50:31.649207 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:31.649165    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls podName:7b82138a-8001-4ce5-80d6-6843f4ced688 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.649142881 +0000 UTC m=+44.628492173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ft7w4" (UID: "7b82138a-8001-4ce5-80d6-6843f4ced688") : secret "samples-operator-tls" not found
Jul 16 21:50:33.163293 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:33.163250    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:33.163665 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.163421    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:33.163665 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.163441    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd5bf47b9-b6s8g: secret "image-registry-tls" not found
Jul 16 21:50:33.163665 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.163491    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls podName:a79de3dd-836d-4018-9482-974abea12707 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.163474938 +0000 UTC m=+52.142824216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls") pod "image-registry-5cd5bf47b9-b6s8g" (UID: "a79de3dd-836d-4018-9482-974abea12707") : secret "image-registry-tls" not found
Jul 16 21:50:33.263681 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:33.263642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:33.263841 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.263798    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:33.263882 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.263878    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls podName:f65c0253-132a-4143-9015-1dd343dd797b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.263862417 +0000 UTC m=+52.243211694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls") pod "dns-default-x9px6" (UID: "f65c0253-132a-4143-9015-1dd343dd797b") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:33.364364 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:33.364328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:33.364537 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.364514    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:33.364663 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.364601    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls podName:37dae182-a8ff-48b8-898e-155bda8bf0c5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.36458004 +0000 UTC m=+48.343929321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-nkrfq" (UID: "37dae182-a8ff-48b8-898e-155bda8bf0c5") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:33.566139 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:33.566046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:33.566290 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.566218    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.566198945 +0000 UTC m=+48.545548222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : configmap references non-existent config key: service-ca.crt
Jul 16 21:50:33.566290 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:33.566284    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:33.566427 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.566376    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:50:33.566427 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.566416    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.56640688 +0000 UTC m=+48.545756157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : secret "router-metrics-certs-default" not found
Jul 16 21:50:33.667172 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:33.667135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:33.667340 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.667302    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:50:33.667399 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:33.667379    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls podName:7b82138a-8001-4ce5-80d6-6843f4ced688 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.667358254 +0000 UTC m=+48.646707531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ft7w4" (UID: "7b82138a-8001-4ce5-80d6-6843f4ced688") : secret "samples-operator-tls" not found
Jul 16 21:50:35.765428 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.765383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf" event={"ID":"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c","Type":"ContainerStarted","Data":"becf091a50091f98c6186dceef866309d30fa5dbb4af030bda6c8f5cf040cef6"}
Jul 16 21:50:35.767356 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.767324    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6" event={"ID":"fb11657a-5fd6-4f2d-8c12-fefc715c6bab","Type":"ContainerStarted","Data":"24a6635c08c42b31ca68eb831e1d77b155bb9b1922470a276a366e19a5bf1b17"}
Jul 16 21:50:35.768915 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.768891    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-t4lc2" event={"ID":"3afdced7-a64f-4cac-858a-362b32874bd5","Type":"ContainerStarted","Data":"51f12f89a71e1342a1cf861e3bd89e5ffbbfd330e8f4ce6e67452aab49b86ea6"}
Jul 16 21:50:35.770593 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.770574    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9" event={"ID":"a3d0824f-6316-40f5-8268-d138a1d9aa61","Type":"ContainerStarted","Data":"f982856fe2bf39dcbca285e962e2bf1fcbd6107fda6b7ea2658b894de160538f"}
Jul 16 21:50:35.772167 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.772148    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/0.log"
Jul 16 21:50:35.772235 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.772186    2557 generic.go:356] "Generic (PLEG): container finished" podID="5af2916e-742e-4253-a404-425b475b1863" containerID="1ef8e7f9c8f6b9673a0c7b9e320f91c863bc4e7c26e065e921dbaebb2dc8e57b" exitCode=255
Jul 16 21:50:35.772285 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.772237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" event={"ID":"5af2916e-742e-4253-a404-425b475b1863","Type":"ContainerDied","Data":"1ef8e7f9c8f6b9673a0c7b9e320f91c863bc4e7c26e065e921dbaebb2dc8e57b"}
Jul 16 21:50:35.772386 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.772373    2557 scope.go:117] "RemoveContainer" containerID="1ef8e7f9c8f6b9673a0c7b9e320f91c863bc4e7c26e065e921dbaebb2dc8e57b"
Jul 16 21:50:35.782915 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.782869    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf" podStartSLOduration=2.121905004 podStartE2EDuration="6.782856138s" totalImagesPullingTime="4.660951134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:35.782086649 +0000 UTC m=+46.761435949" watchObservedRunningTime="2026-07-16 21:50:35.782856138 +0000 UTC m=+46.762205427"
Jul 16 21:50:35.802421 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.802370    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-scjs9" podStartSLOduration=2.045279356 podStartE2EDuration="6.802351558s" totalImagesPullingTime="4.757072202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:35.800933774 +0000 UTC m=+46.780283074" watchObservedRunningTime="2026-07-16 21:50:35.802351558 +0000 UTC m=+46.781700858"
Jul 16 21:50:35.825353 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.824900    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-t4lc2" podStartSLOduration=1.998187276 podStartE2EDuration="6.824881065s" totalImagesPullingTime="4.826693789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:35.824380814 +0000 UTC m=+46.803730116" watchObservedRunningTime="2026-07-16 21:50:35.824881065 +0000 UTC m=+46.804230365"
Jul 16 21:50:35.853427 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:35.853369    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6" podStartSLOduration=2.14088202 podStartE2EDuration="6.853347201s" totalImagesPullingTime="4.712465181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:35.852145292 +0000 UTC m=+46.831494590" watchObservedRunningTime="2026-07-16 21:50:35.853347201 +0000 UTC m=+46.832696501"
Jul 16 21:50:36.653770 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.653735    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"]
Jul 16 21:50:36.674941 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.674914    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"]
Jul 16 21:50:36.675097 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.675026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"
Jul 16 21:50:36.679964 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.679936    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 16 21:50:36.680206 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.680189    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hn8tl\""
Jul 16 21:50:36.680269 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.680242    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:36.776422 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.776403    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 21:50:36.776899 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.776882    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/0.log"
Jul 16 21:50:36.776991 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.776920    2557 generic.go:356] "Generic (PLEG): container finished" podID="5af2916e-742e-4253-a404-425b475b1863" containerID="98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c" exitCode=255
Jul 16 21:50:36.777040 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.777021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" event={"ID":"5af2916e-742e-4253-a404-425b475b1863","Type":"ContainerDied","Data":"98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c"}
Jul 16 21:50:36.777090 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.777074    2557 scope.go:117] "RemoveContainer" containerID="1ef8e7f9c8f6b9673a0c7b9e320f91c863bc4e7c26e065e921dbaebb2dc8e57b"
Jul 16 21:50:36.777199 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.777181    2557 scope.go:117] "RemoveContainer" containerID="98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c"
Jul 16 21:50:36.777422 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:36.777398    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t7tsg_openshift-console-operator(5af2916e-742e-4253-a404-425b475b1863)\"" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" podUID="5af2916e-742e-4253-a404-425b475b1863"
Jul 16 21:50:36.795080 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.795050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvf6\" (UniqueName: \"kubernetes.io/projected/db192757-d00d-4f91-9cff-ce4820bdba31-kube-api-access-jvvf6\") pod \"migrator-7cc9c7ffd7-tzh8s\" (UID: \"db192757-d00d-4f91-9cff-ce4820bdba31\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"
Jul 16 21:50:36.896422 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.896127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvf6\" (UniqueName: \"kubernetes.io/projected/db192757-d00d-4f91-9cff-ce4820bdba31-kube-api-access-jvvf6\") pod \"migrator-7cc9c7ffd7-tzh8s\" (UID: \"db192757-d00d-4f91-9cff-ce4820bdba31\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"
Jul 16 21:50:36.919510 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.919457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvf6\" (UniqueName: \"kubernetes.io/projected/db192757-d00d-4f91-9cff-ce4820bdba31-kube-api-access-jvvf6\") pod \"migrator-7cc9c7ffd7-tzh8s\" (UID: \"db192757-d00d-4f91-9cff-ce4820bdba31\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"
Jul 16 21:50:36.984618 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:36.984596    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"
Jul 16 21:50:37.118339 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.118312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s"]
Jul 16 21:50:37.121640 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:37.121616    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb192757_d00d_4f91_9cff_ce4820bdba31.slice/crio-c50307c924cf2cf0123e3b238d31e4816ea646af5bb0431d2a681c37d61bba5c WatchSource:0}: Error finding container c50307c924cf2cf0123e3b238d31e4816ea646af5bb0431d2a681c37d61bba5c: Status 404 returned error can't find the container with id c50307c924cf2cf0123e3b238d31e4816ea646af5bb0431d2a681c37d61bba5c
Jul 16 21:50:37.399892 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.399858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:37.400039 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.400010    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:37.400086 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.400075    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls podName:37dae182-a8ff-48b8-898e-155bda8bf0c5 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.400059969 +0000 UTC m=+56.379409247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-nkrfq" (UID: "37dae182-a8ff-48b8-898e-155bda8bf0c5") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:37.464131 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.464094    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mh8sz"]
Jul 16 21:50:37.479289 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.479265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.481960 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.481933    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:50:37.482092 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.482055    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q652x\""
Jul 16 21:50:37.482174 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.482087    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:50:37.486072 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.485945    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mh8sz"]
Jul 16 21:50:37.601662 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f06de3dc-1b3f-4823-8c95-9a23dd15889b-data-volume\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.601797 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.601797 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.601932 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:37.601932 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:37.601932 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f06de3dc-1b3f-4823-8c95-9a23dd15889b-crio-socket\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.602021 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.601969    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:50:37.602021 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.601988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqn6\" (UniqueName: \"kubernetes.io/projected/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-api-access-xtqn6\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.602021 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.602009    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.601993055 +0000 UTC m=+56.581342332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : configmap references non-existent config key: service-ca.crt
Jul 16 21:50:37.602115 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.602034    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.602026523 +0000 UTC m=+56.581375800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : secret "router-metrics-certs-default" not found
Jul 16 21:50:37.702451 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702388    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f06de3dc-1b3f-4823-8c95-9a23dd15889b-crio-socket\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702451 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702439    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqn6\" (UniqueName: \"kubernetes.io/projected/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-api-access-xtqn6\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702625 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:37.702625 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f06de3dc-1b3f-4823-8c95-9a23dd15889b-data-volume\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702716 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.702625    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:50:37.702716 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702630    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f06de3dc-1b3f-4823-8c95-9a23dd15889b-crio-socket\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702716 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702716 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.702694    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls podName:7b82138a-8001-4ce5-80d6-6843f4ced688 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.702675052 +0000 UTC m=+56.682024339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ft7w4" (UID: "7b82138a-8001-4ce5-80d6-6843f4ced688") : secret "samples-operator-tls" not found
Jul 16 21:50:37.702927 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702927 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.702792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f06de3dc-1b3f-4823-8c95-9a23dd15889b-data-volume\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.702927 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.702857    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:37.702927 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.702859    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:37.702927 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.702901    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.202890387 +0000 UTC m=+49.182239664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:37.702927 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.702916    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.202906569 +0000 UTC m=+49.182255847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:37.710938 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.710919    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqn6\" (UniqueName: \"kubernetes.io/projected/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-api-access-xtqn6\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:37.780763 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.780734    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s" event={"ID":"db192757-d00d-4f91-9cff-ce4820bdba31","Type":"ContainerStarted","Data":"c50307c924cf2cf0123e3b238d31e4816ea646af5bb0431d2a681c37d61bba5c"}
Jul 16 21:50:37.782332 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.782312    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 21:50:37.782591 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:37.782576    2557 scope.go:117] "RemoveContainer" containerID="98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c"
Jul 16 21:50:37.782730 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:37.782714    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t7tsg_openshift-console-operator(5af2916e-742e-4253-a404-425b475b1863)\"" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" podUID="5af2916e-742e-4253-a404-425b475b1863"
Jul 16 21:50:38.207260 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:38.207232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:38.207387 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:38.207267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:38.207387 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:38.207346    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:38.207387 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:38.207373    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:38.207387 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:38.207390    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:39.207376738 +0000 UTC m=+50.186726015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:38.207540 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:38.207424    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:39.207408424 +0000 UTC m=+50.186757701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:39.216894 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:39.216866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:39.217263 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:39.216916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:39.217263 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:39.216992    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:39.217263 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:39.217042    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:39.217263 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:39.217048    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.217034571 +0000 UTC m=+52.196383858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:39.217263 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:39.217106    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.217090126 +0000 UTC m=+52.196439402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:39.792034 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:39.792000    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s" event={"ID":"db192757-d00d-4f91-9cff-ce4820bdba31","Type":"ContainerStarted","Data":"4e3eefc85220521ed29b49a80889de550580c2cec319646d8776ab12b5106f98"}
Jul 16 21:50:39.792196 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:39.792077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s" event={"ID":"db192757-d00d-4f91-9cff-ce4820bdba31","Type":"ContainerStarted","Data":"f7ad67bd94afb20cf74e77c78bfcf57b3edfc155aee5ef45b9b14ae573085875"}
Jul 16 21:50:40.163749 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:40.163717    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:40.163932 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:40.163827    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:40.164156 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:40.164142    2557 scope.go:117] "RemoveContainer" containerID="98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c"
Jul 16 21:50:40.164304 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:40.164287    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t7tsg_openshift-console-operator(5af2916e-742e-4253-a404-425b475b1863)\"" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" podUID="5af2916e-742e-4253-a404-425b475b1863"
Jul 16 21:50:40.458290 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:40.458216    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jkm6h_fb1570a9-59ea-4345-8408-5a192d1e24c6/dns-node-resolver/0.log"
Jul 16 21:50:40.795219 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:40.795148    2557 scope.go:117] "RemoveContainer" containerID="98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c"
Jul 16 21:50:40.795356 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:40.795293    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t7tsg_openshift-console-operator(5af2916e-742e-4253-a404-425b475b1863)\"" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" podUID="5af2916e-742e-4253-a404-425b475b1863"
Jul 16 21:50:41.061132 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:41.061063    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6z6pz_52d062bf-25da-41c5-b1ae-0ecd872eaedf/node-ca/0.log"
Jul 16 21:50:41.232491 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:41.232458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:41.232491 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:41.232491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:41.232689 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.232619    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:41.232724 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.232695    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.232677362 +0000 UTC m=+56.212026643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:41.232724 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:41.232688    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:41.232797 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.232760    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:41.232797 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.232772    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd5bf47b9-b6s8g: secret "image-registry-tls" not found
Jul 16 21:50:41.232885 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.232848    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls podName:a79de3dd-836d-4018-9482-974abea12707 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:57.232834508 +0000 UTC m=+68.212183808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls") pod "image-registry-5cd5bf47b9-b6s8g" (UID: "a79de3dd-836d-4018-9482-974abea12707") : secret "image-registry-tls" not found
Jul 16 21:50:41.233021 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:41.233005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f06de3dc-1b3f-4823-8c95-9a23dd15889b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:41.334153 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:41.334078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:41.334289 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.334241    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:41.334326 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:41.334301    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls podName:f65c0253-132a-4143-9015-1dd343dd797b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:57.334282925 +0000 UTC m=+68.313632223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls") pod "dns-default-x9px6" (UID: "f65c0253-132a-4143-9015-1dd343dd797b") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:42.055319 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:42.055290    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8rcrk_97aa7d52-d173-4110-90be-35f610bdda8f/serve-healthcheck-canary/0.log"
Jul 16 21:50:42.454775 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:42.454745    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tzh8s_db192757-d00d-4f91-9cff-ce4820bdba31/migrator/0.log"
Jul 16 21:50:42.654533 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:42.654504    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tzh8s_db192757-d00d-4f91-9cff-ce4820bdba31/graceful-termination/0.log"
Jul 16 21:50:42.855864 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:42.855781    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-vtvvf_8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c/kube-storage-version-migrator-operator/0.log"
Jul 16 21:50:45.269252 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.269205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:45.269715 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.269360    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:45.269715 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.269432    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls podName:f06de3dc-1b3f-4823-8c95-9a23dd15889b nodeName:}" failed. No retries permitted until 2026-07-16 21:50:53.269416987 +0000 UTC m=+64.248766265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mh8sz" (UID: "f06de3dc-1b3f-4823-8c95-9a23dd15889b") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:45.470592 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.470549    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:50:45.470769 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.470701    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:45.470769 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.470767    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls podName:37dae182-a8ff-48b8-898e-155bda8bf0c5 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:01.470752262 +0000 UTC m=+72.450101542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-nkrfq" (UID: "37dae182-a8ff-48b8-898e-155bda8bf0c5") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:45.672142 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.672100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:45.672142 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.672146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:50:45.672385 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.672250    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:50:45.672385 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.672304    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:01.672285875 +0000 UTC m=+72.651635154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : configmap references non-existent config key: service-ca.crt
Jul 16 21:50:45.672385 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:50:45.672324    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs podName:ec73e762-8ce7-42b8-b8b4-bad1d863c480 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:01.672315573 +0000 UTC m=+72.651664856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs") pod "router-default-559484955d-q46dq" (UID: "ec73e762-8ce7-42b8-b8b4-bad1d863c480") : secret "router-metrics-certs-default" not found
Jul 16 21:50:45.773066 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.773025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:45.775516 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.775496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b82138a-8001-4ce5-80d6-6843f4ced688-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ft7w4\" (UID: \"7b82138a-8001-4ce5-80d6-6843f4ced688\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:45.811308 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.811260    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"
Jul 16 21:50:45.935197 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.935097    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tzh8s" podStartSLOduration=7.985502627 podStartE2EDuration="9.935073091s" totalImagesPullingTime="1.949570464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:39.819152294 +0000 UTC m=+50.798501605" watchObservedRunningTime="2026-07-16 21:50:45.935073091 +0000 UTC m=+56.914422391"
Jul 16 21:50:45.935623 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:45.935605    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4"]
Jul 16 21:50:46.819938 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:46.819899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4" event={"ID":"7b82138a-8001-4ce5-80d6-6843f4ced688","Type":"ContainerStarted","Data":"5fe85ef413b586fe18d9fce3c3f37ff99bbf03646bdbe8097a0a9aaec592b82a"}
Jul 16 21:50:47.824778 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:47.824748    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4" event={"ID":"7b82138a-8001-4ce5-80d6-6843f4ced688","Type":"ContainerStarted","Data":"ff0ee91ad4d6de1126c6521cd80db0567f0d8a2c2e060c49e093f583d03c3854"}
Jul 16 21:50:47.825147 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:47.824796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4" event={"ID":"7b82138a-8001-4ce5-80d6-6843f4ced688","Type":"ContainerStarted","Data":"ece9bce0604a806c82bcea84f77900323a8b30665064d0fc47551c6496abd587"}
Jul 16 21:50:47.841309 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:47.841250    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ft7w4" podStartSLOduration=17.211002369 podStartE2EDuration="18.841232111s" totalImagesPullingTime="1.630229742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:47.839919733 +0000 UTC m=+58.819269033" watchObservedRunningTime="2026-07-16 21:50:47.841232111 +0000 UTC m=+58.820581410"
Jul 16 21:50:50.713246 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:50.713214    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k6b2"
Jul 16 21:50:53.338470 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.338429    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:53.340857 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.340834    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f06de3dc-1b3f-4823-8c95-9a23dd15889b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mh8sz\" (UID: \"f06de3dc-1b3f-4823-8c95-9a23dd15889b\") " pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:53.392926 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.392898    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q652x\""
Jul 16 21:50:53.398996 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.398979    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mh8sz"
Jul 16 21:50:53.525596 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.525568    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mh8sz"]
Jul 16 21:50:53.528711 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:53.528682    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06de3dc_1b3f_4823_8c95_9a23dd15889b.slice/crio-53f857ab818a105260bfa476b30c705f5a9d40a421f8fa9d59a8f6c59a61d6e8 WatchSource:0}: Error finding container 53f857ab818a105260bfa476b30c705f5a9d40a421f8fa9d59a8f6c59a61d6e8: Status 404 returned error can't find the container with id 53f857ab818a105260bfa476b30c705f5a9d40a421f8fa9d59a8f6c59a61d6e8
Jul 16 21:50:53.557144 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.557127    2557 scope.go:117] "RemoveContainer" containerID="98c7afda1e7fb7521517e3494c154260857c841360842ede06f932b46df4e07c"
Jul 16 21:50:53.843229 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.843149    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 21:50:53.843365 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.843266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" event={"ID":"5af2916e-742e-4253-a404-425b475b1863","Type":"ContainerStarted","Data":"50b2d0b0cf500e2535a7b9ee6e8ad64f797a49b95018bfee852c2a97acae7b2e"}
Jul 16 21:50:53.843627 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.843591    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:53.844696 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.844669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mh8sz" event={"ID":"f06de3dc-1b3f-4823-8c95-9a23dd15889b","Type":"ContainerStarted","Data":"53f857ab818a105260bfa476b30c705f5a9d40a421f8fa9d59a8f6c59a61d6e8"}
Jul 16 21:50:53.863460 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:53.863407    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg" podStartSLOduration=20.17471083 podStartE2EDuration="24.863386777s" totalImagesPullingTime="4.688675947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:53.862651959 +0000 UTC m=+64.842001260" watchObservedRunningTime="2026-07-16 21:50:53.863386777 +0000 UTC m=+64.842736069"
Jul 16 21:50:54.832773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:54.832739    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-t7tsg"
Jul 16 21:50:55.259582 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.259544    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:55.259749 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.259601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:55.259749 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.259711    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:55.264433 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.264406    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:50:55.264433 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.264428    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:50:55.264611 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.264409    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:50:55.273361 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.273337    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2-original-pull-secret\") pod \"global-pull-secret-syncer-cpdkm\" (UID: \"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2\") " pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:55.273361 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.273357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24fdc43a-8826-46d0-8c18-ace05937ce99-metrics-certs\") pod \"network-metrics-daemon-dpnxv\" (UID: \"24fdc43a-8826-46d0-8c18-ace05937ce99\") " pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:55.274046 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.274024    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:50:55.284045 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.284025    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmmh\" (UniqueName: \"kubernetes.io/projected/8c1eb6f4-cde0-4dad-a483-e8e265add7ce-kube-api-access-ktmmh\") pod \"network-check-target-sr4st\" (UID: \"8c1eb6f4-cde0-4dad-a483-e8e265add7ce\") " pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:55.372017 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.371984    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn8m8\""
Jul 16 21:50:55.376652 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.376628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpdkm"
Jul 16 21:50:55.379737 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.379703    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpnxv"
Jul 16 21:50:55.385071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.385037    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qpnd8\""
Jul 16 21:50:55.392905 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.392879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:50:55.565030 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.564993    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cpdkm"]
Jul 16 21:50:55.568044 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:55.568003    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f90b1b1_28ae_4fda_972d_b2be4f1ae0c2.slice/crio-1b19db35c45b03d330435cb8c75264c00cdc3ba0ebee0ad9ea61e1a98800ae89 WatchSource:0}: Error finding container 1b19db35c45b03d330435cb8c75264c00cdc3ba0ebee0ad9ea61e1a98800ae89: Status 404 returned error can't find the container with id 1b19db35c45b03d330435cb8c75264c00cdc3ba0ebee0ad9ea61e1a98800ae89
Jul 16 21:50:55.583794 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.583763    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dpnxv"]
Jul 16 21:50:55.587533 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:55.587502    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fdc43a_8826_46d0_8c18_ace05937ce99.slice/crio-770cd59a8571d71bdda2a37f31941d9529f68689537341ea00a1996211600e9e WatchSource:0}: Error finding container 770cd59a8571d71bdda2a37f31941d9529f68689537341ea00a1996211600e9e: Status 404 returned error can't find the container with id 770cd59a8571d71bdda2a37f31941d9529f68689537341ea00a1996211600e9e
Jul 16 21:50:55.608286 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:50:55.607386    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1eb6f4_cde0_4dad_a483_e8e265add7ce.slice/crio-da05d169367bcbc327fc62f350b648f495af553c90097dfb94ee8d35d9703967 WatchSource:0}: Error finding container da05d169367bcbc327fc62f350b648f495af553c90097dfb94ee8d35d9703967: Status 404 returned error can't find the container with id da05d169367bcbc327fc62f350b648f495af553c90097dfb94ee8d35d9703967
Jul 16 21:50:55.608286 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.608050    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sr4st"]
Jul 16 21:50:55.851799 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.851712    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpnxv" event={"ID":"24fdc43a-8826-46d0-8c18-ace05937ce99","Type":"ContainerStarted","Data":"770cd59a8571d71bdda2a37f31941d9529f68689537341ea00a1996211600e9e"}
Jul 16 21:50:55.852875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.852846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cpdkm" event={"ID":"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2","Type":"ContainerStarted","Data":"1b19db35c45b03d330435cb8c75264c00cdc3ba0ebee0ad9ea61e1a98800ae89"}
Jul 16 21:50:55.854394 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:55.854369    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sr4st" event={"ID":"8c1eb6f4-cde0-4dad-a483-e8e265add7ce","Type":"ContainerStarted","Data":"da05d169367bcbc327fc62f350b648f495af553c90097dfb94ee8d35d9703967"}
Jul 16 21:50:57.277250 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.277199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:57.280050 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.280024    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"image-registry-5cd5bf47b9-b6s8g\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") " pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:57.378007 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.377979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:57.380600 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.380572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f65c0253-132a-4143-9015-1dd343dd797b-metrics-tls\") pod \"dns-default-x9px6\" (UID: \"f65c0253-132a-4143-9015-1dd343dd797b\") " pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:57.540662 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.540581    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-47gk2\""
Jul 16 21:50:57.548649 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.548618    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:57.582698 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.582675    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g56tn\""
Jul 16 21:50:57.590952 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:57.590931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9px6"
Jul 16 21:50:59.330561 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.330484    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"]
Jul 16 21:50:59.343642 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.343611    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9px6"]
Jul 16 21:50:59.872669 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.872593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g" event={"ID":"a79de3dd-836d-4018-9482-974abea12707","Type":"ContainerStarted","Data":"f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525"}
Jul 16 21:50:59.872884 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.872701    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:50:59.872884 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.872725    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g" event={"ID":"a79de3dd-836d-4018-9482-974abea12707","Type":"ContainerStarted","Data":"1cf07eb5f3084bac5c167cbb6308a354811fec3769f433271139e0276bf87941"}
Jul 16 21:50:59.877652 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.877021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpnxv" event={"ID":"24fdc43a-8826-46d0-8c18-ace05937ce99","Type":"ContainerStarted","Data":"a6736f0f8e0c0148214b210ddcb33b98f9656761ddd7e0476a501c85d7bb51bd"}
Jul 16 21:50:59.877652 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.877081    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpnxv" event={"ID":"24fdc43a-8826-46d0-8c18-ace05937ce99","Type":"ContainerStarted","Data":"1e493fd8f1da50d5d81452009986c139510405022ba21ba650d494c58f1a319f"}
Jul 16 21:50:59.880119 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.880033    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mh8sz" event={"ID":"f06de3dc-1b3f-4823-8c95-9a23dd15889b","Type":"ContainerStarted","Data":"c5bf556bb4cce3aa8aed857cfe8f31c1b3fe5889da890e1fe05ad910a46c83c8"}
Jul 16 21:50:59.881739 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.881710    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9px6" event={"ID":"f65c0253-132a-4143-9015-1dd343dd797b","Type":"ContainerStarted","Data":"c37bb8d6846676924e8230d9ad7ae2651c8f2a911ae971a2836bf37a13597623"}
Jul 16 21:50:59.895279 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.895216    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g" podStartSLOduration=70.895200082 podStartE2EDuration="1m10.895200082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:59.894283151 +0000 UTC m=+70.873632463" watchObservedRunningTime="2026-07-16 21:50:59.895200082 +0000 UTC m=+70.874549382"
Jul 16 21:50:59.911272 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:50:59.911216    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dpnxv" podStartSLOduration=66.956955769 podStartE2EDuration="1m10.911200429s" totalImagesPullingTime="3.95424466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:59.910392485 +0000 UTC m=+70.889741784" watchObservedRunningTime="2026-07-16 21:50:59.911200429 +0000 UTC m=+70.890549729"
Jul 16 21:51:01.518269 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.518233    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:51:01.521437 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.521396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37dae182-a8ff-48b8-898e-155bda8bf0c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-nkrfq\" (UID: \"37dae182-a8ff-48b8-898e-155bda8bf0c5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:51:01.719769 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.719636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:01.719769 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.719687    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:01.720516 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.720487    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec73e762-8ce7-42b8-b8b4-bad1d863c480-service-ca-bundle\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:01.722804 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.722780    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec73e762-8ce7-42b8-b8b4-bad1d863c480-metrics-certs\") pod \"router-default-559484955d-q46dq\" (UID: \"ec73e762-8ce7-42b8-b8b4-bad1d863c480\") " pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:01.731941 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.731917    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dtn4x\""
Jul 16 21:51:01.738860 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.738804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"
Jul 16 21:51:01.881918 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.881849    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xtb2b\""
Jul 16 21:51:01.889251 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:01.889224    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:05.823396 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:05.823365    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq"]
Jul 16 21:51:05.868679 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:51:05.868306    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dae182_a8ff_48b8_898e_155bda8bf0c5.slice/crio-c5b2f7482c5a47f92b7df8d41a429060d7fda8fae167223f244e4a5c2679be9f WatchSource:0}: Error finding container c5b2f7482c5a47f92b7df8d41a429060d7fda8fae167223f244e4a5c2679be9f: Status 404 returned error can't find the container with id c5b2f7482c5a47f92b7df8d41a429060d7fda8fae167223f244e4a5c2679be9f
Jul 16 21:51:05.906133 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:05.906102    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq" event={"ID":"37dae182-a8ff-48b8-898e-155bda8bf0c5","Type":"ContainerStarted","Data":"c5b2f7482c5a47f92b7df8d41a429060d7fda8fae167223f244e4a5c2679be9f"}
Jul 16 21:51:06.023795 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.023771    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-559484955d-q46dq"]
Jul 16 21:51:06.026792 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:51:06.026767    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec73e762_8ce7_42b8_b8b4_bad1d863c480.slice/crio-81599e29ed3731337ed9e8b6722c0aac0217175c587ab5ff63e02157d2cb1a34 WatchSource:0}: Error finding container 81599e29ed3731337ed9e8b6722c0aac0217175c587ab5ff63e02157d2cb1a34: Status 404 returned error can't find the container with id 81599e29ed3731337ed9e8b6722c0aac0217175c587ab5ff63e02157d2cb1a34
Jul 16 21:51:06.912793 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.912753    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sr4st" event={"ID":"8c1eb6f4-cde0-4dad-a483-e8e265add7ce","Type":"ContainerStarted","Data":"9670841c8861c3e3977d19a75f64cbbae07abe78b1962b97ceedc999d804faf0"}
Jul 16 21:51:06.913548 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.913524    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:51:06.915880 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.915853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mh8sz" event={"ID":"f06de3dc-1b3f-4823-8c95-9a23dd15889b","Type":"ContainerStarted","Data":"e04c8a94cd720d38fdcfd08278753e6a078d64bf11c718421ef148e1922c67cd"}
Jul 16 21:51:06.918390 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.918363    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cpdkm" event={"ID":"4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2","Type":"ContainerStarted","Data":"5647d6391c90bf619aaf6842a94bb7aa17bcf92e46e3abd85131a82abfb7455c"}
Jul 16 21:51:06.922975 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.922940    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9px6" event={"ID":"f65c0253-132a-4143-9015-1dd343dd797b","Type":"ContainerStarted","Data":"6d50e380756220340885750bf9f3196a1121f3c10be943dd121b78263d1da54c"}
Jul 16 21:51:06.923075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.922987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9px6" event={"ID":"f65c0253-132a-4143-9015-1dd343dd797b","Type":"ContainerStarted","Data":"9da86dcfb088ea4c209c58d7a49b501e91d9f6aa1f4bab18917a2ad59aab86f5"}
Jul 16 21:51:06.923146 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.923074    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x9px6"
Jul 16 21:51:06.926250 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.925826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-559484955d-q46dq" event={"ID":"ec73e762-8ce7-42b8-b8b4-bad1d863c480","Type":"ContainerStarted","Data":"c2332b1f81e2879eaf39892de8f31fadc1d616beb51beb2495c8df49a865deeb"}
Jul 16 21:51:06.926250 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.925864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-559484955d-q46dq" event={"ID":"ec73e762-8ce7-42b8-b8b4-bad1d863c480","Type":"ContainerStarted","Data":"81599e29ed3731337ed9e8b6722c0aac0217175c587ab5ff63e02157d2cb1a34"}
Jul 16 21:51:06.940748 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.940707    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sr4st" podStartSLOduration=67.88263198 podStartE2EDuration="1m17.940694858s" totalImagesPullingTime="10.058062878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:06.940400314 +0000 UTC m=+77.919749615" watchObservedRunningTime="2026-07-16 21:51:06.940694858 +0000 UTC m=+77.920044160"
Jul 16 21:51:06.967613 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.967568    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x9px6" podStartSLOduration=35.616990128 podStartE2EDuration="41.967554479s" totalImagesPullingTime="6.350564351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:06.965984442 +0000 UTC m=+77.945333742" watchObservedRunningTime="2026-07-16 21:51:06.967554479 +0000 UTC m=+77.946903780"
Jul 16 21:51:06.987242 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:06.987199    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-559484955d-q46dq" podStartSLOduration=37.987182911 podStartE2EDuration="37.987182911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:06.986235555 +0000 UTC m=+77.965584856" watchObservedRunningTime="2026-07-16 21:51:06.987182911 +0000 UTC m=+77.966532211"
Jul 16 21:51:07.007765 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:07.007723    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cpdkm" podStartSLOduration=67.707801329 podStartE2EDuration="1m18.007709138s" totalImagesPullingTime="10.299907809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:07.007086315 +0000 UTC m=+77.986435618" watchObservedRunningTime="2026-07-16 21:51:07.007709138 +0000 UTC m=+77.987058438"
Jul 16 21:51:07.889687 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:07.889652    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:07.892479 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:07.892446    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:07.931488 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:07.931455    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:07.932970 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:07.932826    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-559484955d-q46dq"
Jul 16 21:51:08.937684 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:08.937629    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq" event={"ID":"37dae182-a8ff-48b8-898e-155bda8bf0c5","Type":"ContainerStarted","Data":"f64c08e856015b86c76bb3b3b53fedeab2f2f1a69b2da8ccdd6c426a2775c072"}
Jul 16 21:51:08.968537 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:08.968474    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq" podStartSLOduration=37.769338734 podStartE2EDuration="39.968452263s" totalImagesPullingTime="2.199113529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:08.967389927 +0000 UTC m=+79.946739242" watchObservedRunningTime="2026-07-16 21:51:08.968452263 +0000 UTC m=+79.947801563"
Jul 16 21:51:10.947160 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:10.947126    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mh8sz" event={"ID":"f06de3dc-1b3f-4823-8c95-9a23dd15889b","Type":"ContainerStarted","Data":"5eb17b7eeb28c1b0f0e9073fe9b445e14249ed3475e0b532580b035111333606"}
Jul 16 21:51:10.967844 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:10.967779    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mh8sz" podStartSLOduration=17.641604516 podStartE2EDuration="33.967763401s" totalImagesPullingTime="16.326158885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:10.967152354 +0000 UTC m=+81.946501687" watchObservedRunningTime="2026-07-16 21:51:10.967763401 +0000 UTC m=+81.947112700"
Jul 16 21:51:16.934086 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:16.934058    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x9px6"
Jul 16 21:51:20.709910 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:20.709870    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"]
Jul 16 21:51:20.717847 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:20.717804    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:51:31.444543 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.444514    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d26k5"]
Jul 16 21:51:31.615723 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.615698    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.619517 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.619485    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:51:31.620106 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.620088    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:51:31.620451 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.620179    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:51:31.620648 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.620631    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l45wb\""
Jul 16 21:51:31.622016 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.620686    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:51:31.758505 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758505 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklgd\" (UniqueName: \"kubernetes.io/projected/e792edf0-10d2-409d-9702-ef265c84ba4b-kube-api-access-lklgd\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758505 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-root\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758671 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758585    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-tls\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758671 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758610    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e792edf0-10d2-409d-9702-ef265c84ba4b-metrics-client-ca\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758671 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758762 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-wtmp\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758762 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-textfile\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.758854 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.758770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-sys\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859435 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859410    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859531 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklgd\" (UniqueName: \"kubernetes.io/projected/e792edf0-10d2-409d-9702-ef265c84ba4b-kube-api-access-lklgd\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859531 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859462    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-root\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859531 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-tls\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859531 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e792edf0-10d2-409d-9702-ef265c84ba4b-metrics-client-ca\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859531 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859512    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-root\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-wtmp\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-textfile\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-sys\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859702    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-wtmp\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.859773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.859704    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e792edf0-10d2-409d-9702-ef265c84ba4b-sys\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.860080 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.860057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-textfile\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.860196 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.860154    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.860196 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.860193    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e792edf0-10d2-409d-9702-ef265c84ba4b-metrics-client-ca\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.861929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.861910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.862137 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.862115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e792edf0-10d2-409d-9702-ef265c84ba4b-node-exporter-tls\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.871320 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.871296    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklgd\" (UniqueName: \"kubernetes.io/projected/e792edf0-10d2-409d-9702-ef265c84ba4b-kube-api-access-lklgd\") pod \"node-exporter-d26k5\" (UID: \"e792edf0-10d2-409d-9702-ef265c84ba4b\") " pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.927398 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:31.927374    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d26k5"
Jul 16 21:51:31.966985 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:51:31.966964    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode792edf0_10d2_409d_9702_ef265c84ba4b.slice/crio-2c4c2bd21736e3b554685cca9d8cb8b695e9124b21d9034e1abe94dd26df46ae WatchSource:0}: Error finding container 2c4c2bd21736e3b554685cca9d8cb8b695e9124b21d9034e1abe94dd26df46ae: Status 404 returned error can't find the container with id 2c4c2bd21736e3b554685cca9d8cb8b695e9124b21d9034e1abe94dd26df46ae
Jul 16 21:51:32.019470 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.019416    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d26k5" event={"ID":"e792edf0-10d2-409d-9702-ef265c84ba4b","Type":"ContainerStarted","Data":"2c4c2bd21736e3b554685cca9d8cb8b695e9124b21d9034e1abe94dd26df46ae"}
Jul 16 21:51:32.416911 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.416882    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:51:32.445327 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.445302    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:51:32.445676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.445450    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.448540 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.448522    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 21:51:32.449128 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.449111    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 21:51:32.449416 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.449390    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 21:51:32.450219 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.450199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 21:51:32.450483 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.450459    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 21:51:32.450685 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.450639    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8np2r\""
Jul 16 21:51:32.450685 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.450647    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 21:51:32.451134 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.451114    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 21:51:32.451215 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.451192    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 21:51:32.458580 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.458559    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 21:51:32.566484 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566456    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566524    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwg52\" (UniqueName: \"kubernetes.io/projected/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-kube-api-access-cwg52\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566839 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566839 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566953 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-config-out\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566953 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-web-config\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566953 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.566953 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.567140 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.566969    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.567140 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.567017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.667838 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.667756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-config-out\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.667838 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.667792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-web-config\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.667838 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.667824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668031 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.667972    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668087 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668138 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668088    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668190 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668190 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668193    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg52\" (UniqueName: \"kubernetes.io/projected/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-kube-api-access-cwg52\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668265    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.668299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.668297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.669469 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.669204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.669583 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.669476    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.671906 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.671863    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.672027 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.671960    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.672390 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.672367    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.672478 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.672394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-web-config\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.672478 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.672429    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.672959 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.672936    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.673054 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.672990    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.673392 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.673371    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-config-out\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.674290 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.674271    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.677283 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.677262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwg52\" (UniqueName: \"kubernetes.io/projected/55099ce9-fbdd-4d73-9d22-9b95cf5700a2-kube-api-access-cwg52\") pod \"alertmanager-main-0\" (UID: \"55099ce9-fbdd-4d73-9d22-9b95cf5700a2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.757032 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.757008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:51:32.919277 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:32.919086    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:51:33.125967 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:51:33.125938    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55099ce9_fbdd_4d73_9d22_9b95cf5700a2.slice/crio-39e79a0c7edfc3606ce42f85462f3c415e853f9bd6d8cb6318de8911e4176697 WatchSource:0}: Error finding container 39e79a0c7edfc3606ce42f85462f3c415e853f9bd6d8cb6318de8911e4176697: Status 404 returned error can't find the container with id 39e79a0c7edfc3606ce42f85462f3c415e853f9bd6d8cb6318de8911e4176697
Jul 16 21:51:34.034079 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:34.033994    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"39e79a0c7edfc3606ce42f85462f3c415e853f9bd6d8cb6318de8911e4176697"}
Jul 16 21:51:34.035845 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:34.035794    2557 generic.go:356] "Generic (PLEG): container finished" podID="e792edf0-10d2-409d-9702-ef265c84ba4b" containerID="2a945f64604d828a85da4c675d06bb2a1a929fbff013d0b33a55bbf8f2059a8f" exitCode=0
Jul 16 21:51:34.035954 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:34.035854    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d26k5" event={"ID":"e792edf0-10d2-409d-9702-ef265c84ba4b","Type":"ContainerDied","Data":"2a945f64604d828a85da4c675d06bb2a1a929fbff013d0b33a55bbf8f2059a8f"}
Jul 16 21:51:35.041245 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:35.041217    2557 generic.go:356] "Generic (PLEG): container finished" podID="55099ce9-fbdd-4d73-9d22-9b95cf5700a2" containerID="fad29720e17d2389408d3abd19c6205c7c62ded17e973c222713410c8273acd2" exitCode=0
Jul 16 21:51:35.041644 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:35.041297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerDied","Data":"fad29720e17d2389408d3abd19c6205c7c62ded17e973c222713410c8273acd2"}
Jul 16 21:51:35.043503 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:35.043475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d26k5" event={"ID":"e792edf0-10d2-409d-9702-ef265c84ba4b","Type":"ContainerStarted","Data":"144697ebc99e796154e5475c7ab345da0f0d10bc901878e76e5353b2a2555954"}
Jul 16 21:51:35.043581 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:35.043513    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d26k5" event={"ID":"e792edf0-10d2-409d-9702-ef265c84ba4b","Type":"ContainerStarted","Data":"5ea9a67abaa3f44536b70ce1aa95f3ab6bd730b7b31c63f46adbd187060bd343"}
Jul 16 21:51:35.134596 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:35.134554    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d26k5" podStartSLOduration=2.519117271 podStartE2EDuration="4.1345403s" totalImagesPullingTime="1.507675106s" totalInitContainerRuntime="107.747923ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:35.134318886 +0000 UTC m=+106.113668187" watchObservedRunningTime="2026-07-16 21:51:35.1345403 +0000 UTC m=+106.113889598"
Jul 16 21:51:36.688213 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.688181    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-856bc5cddb-psgm6"]
Jul 16 21:51:36.722602 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.722568    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-856bc5cddb-psgm6"]
Jul 16 21:51:36.722742 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.722690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.725384 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.725359    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 16 21:51:36.725532 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.725423    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 16 21:51:36.725532 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.725359    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 16 21:51:36.725532 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.725361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 16 21:51:36.725788 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.725775    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-j8b7h\""
Jul 16 21:51:36.726029 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.726014    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 16 21:51:36.734059 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.734037    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 16 21:51:36.801690 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801667    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-metrics-client-ca\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-serving-certs-ca-bundle\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swz7\" (UniqueName: \"kubernetes.io/projected/204de874-c30e-409f-849f-2240ca82c63b-kube-api-access-9swz7\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801959 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-secret-telemeter-client\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801959 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801907    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-telemeter-client-tls\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.801959 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.801943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-federate-client-tls\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902546 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-serving-certs-ca-bundle\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swz7\" (UniqueName: \"kubernetes.io/projected/204de874-c30e-409f-849f-2240ca82c63b-kube-api-access-9swz7\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902593    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-secret-telemeter-client\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902628 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902617    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-telemeter-client-tls\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-federate-client-tls\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-metrics-client-ca\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.902792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.902718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.903342 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.903319    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-serving-certs-ca-bundle\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.903636 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.903386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-metrics-client-ca\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.903636 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.903546    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204de874-c30e-409f-849f-2240ca82c63b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.905416 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.905364    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-telemeter-client-tls\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.905514 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.905478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.905569 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.905537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-federate-client-tls\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.905871 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.905844    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/204de874-c30e-409f-849f-2240ca82c63b-secret-telemeter-client\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:36.911004 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:36.910980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swz7\" (UniqueName: \"kubernetes.io/projected/204de874-c30e-409f-849f-2240ca82c63b-kube-api-access-9swz7\") pod \"telemeter-client-856bc5cddb-psgm6\" (UID: \"204de874-c30e-409f-849f-2240ca82c63b\") " pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:37.032712 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:37.032664    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6"
Jul 16 21:51:37.206512 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:37.206466    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-856bc5cddb-psgm6"]
Jul 16 21:51:37.221879 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:51:37.221853    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204de874_c30e_409f_849f_2240ca82c63b.slice/crio-a735ec587c30a0be063afe8bf3e92e536dd1c689b06ead04630f6f0417bd88a4 WatchSource:0}: Error finding container a735ec587c30a0be063afe8bf3e92e536dd1c689b06ead04630f6f0417bd88a4: Status 404 returned error can't find the container with id a735ec587c30a0be063afe8bf3e92e536dd1c689b06ead04630f6f0417bd88a4
Jul 16 21:51:38.062459 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.062422    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"41b090c63e889984afbcf232f698d82b312dcd1575e1ca5022438cfbb1510b5d"}
Jul 16 21:51:38.062964 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.062506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"3473e0f87c14c7dfec782fd52c838859f937ccd3ce58d359723c79583cb9cca0"}
Jul 16 21:51:38.062964 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.062545    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"df893c021ef45e52e94ac74f2ebc0af1729b6d85fb399860f62072dc228f35b0"}
Jul 16 21:51:38.062964 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.062584    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"7e3f9af275163faa626e6472009ed06b116a3ab97c71045a74c153824cd05214"}
Jul 16 21:51:38.062964 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.062621    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"b7bfab2cdf8731c62dadb36a129cd67e3be174c2148ff9bc0d8993ddf5c3dce9"}
Jul 16 21:51:38.063620 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.063593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6" event={"ID":"204de874-c30e-409f-849f-2240ca82c63b","Type":"ContainerStarted","Data":"a735ec587c30a0be063afe8bf3e92e536dd1c689b06ead04630f6f0417bd88a4"}
Jul 16 21:51:38.940480 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:38.940449    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sr4st"
Jul 16 21:51:39.072142 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:39.072070    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55099ce9-fbdd-4d73-9d22-9b95cf5700a2","Type":"ContainerStarted","Data":"6af526a0154b130e17b62891853491092b06308a4b33489f632b05fa1e45fc94"}
Jul 16 21:51:39.121189 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:39.121127    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.466271377 podStartE2EDuration="7.121104643s" totalImagesPullingTime="4.633011884s" totalInitContainerRuntime="21.821382ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:39.120115568 +0000 UTC m=+110.099464868" watchObservedRunningTime="2026-07-16 21:51:39.121104643 +0000 UTC m=+110.100453947"
Jul 16 21:51:40.077750 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:40.077717    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6" event={"ID":"204de874-c30e-409f-849f-2240ca82c63b","Type":"ContainerStarted","Data":"0177241f2c1bfa0821f0ddb59e77d6090de9987a2851db0554fe4b681d683d35"}
Jul 16 21:51:40.078106 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:40.077803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6" event={"ID":"204de874-c30e-409f-849f-2240ca82c63b","Type":"ContainerStarted","Data":"14ff44377af1f2511557484bdb0195c39b25d2c7dcde2ec2126377e626fb6988"}
Jul 16 21:51:40.078106 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:40.077845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6" event={"ID":"204de874-c30e-409f-849f-2240ca82c63b","Type":"ContainerStarted","Data":"543c3e152c1d22512169876127cca25af0975a7452d039a715d2d51ae30a95c9"}
Jul 16 21:51:40.111662 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:40.111370    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-856bc5cddb-psgm6" podStartSLOduration=1.639125671 podStartE2EDuration="4.111348964s" totalImagesPullingTime="2.472223293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:40.110014899 +0000 UTC m=+111.089364238" watchObservedRunningTime="2026-07-16 21:51:40.111348964 +0000 UTC m=+111.090698265"
Jul 16 21:51:45.733924 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:45.733880    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g" podUID="a79de3dd-836d-4018-9482-974abea12707" containerName="registry" containerID="cri-o://f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525" gracePeriod=30
Jul 16 21:51:45.977616 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:45.977593    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:51:46.071649 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071589    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-bound-sa-token\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071649 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071621    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a79de3dd-836d-4018-9482-974abea12707-ca-trust-extracted\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071649 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071650    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071903 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071670    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-trusted-ca\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071903 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071712    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-installation-pull-secrets\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071903 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxm25\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-kube-api-access-wxm25\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071903 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071773    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-registry-certificates\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.071903 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.071882    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-image-registry-private-configuration\") pod \"a79de3dd-836d-4018-9482-974abea12707\" (UID: \"a79de3dd-836d-4018-9482-974abea12707\") "
Jul 16 21:51:46.072233 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.072203    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:46.072547 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.072522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:46.074347 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.074305    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:46.074460 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.074393    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:46.074689 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.074660    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:46.074749 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.074668    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:46.074749 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.074714    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-kube-api-access-wxm25" (OuterVolumeSpecName: "kube-api-access-wxm25") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "kube-api-access-wxm25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:46.080558 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.080534    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79de3dd-836d-4018-9482-974abea12707-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a79de3dd-836d-4018-9482-974abea12707" (UID: "a79de3dd-836d-4018-9482-974abea12707"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:51:46.101036 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.101010    2557 generic.go:356] "Generic (PLEG): container finished" podID="a79de3dd-836d-4018-9482-974abea12707" containerID="f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525" exitCode=0
Jul 16 21:51:46.101110 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.101052    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"
Jul 16 21:51:46.101110 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.101089    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g" event={"ID":"a79de3dd-836d-4018-9482-974abea12707","Type":"ContainerDied","Data":"f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525"}
Jul 16 21:51:46.101185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.101138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd5bf47b9-b6s8g" event={"ID":"a79de3dd-836d-4018-9482-974abea12707","Type":"ContainerDied","Data":"1cf07eb5f3084bac5c167cbb6308a354811fec3769f433271139e0276bf87941"}
Jul 16 21:51:46.101185 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.101159    2557 scope.go:117] "RemoveContainer" containerID="f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525"
Jul 16 21:51:46.109415 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.109397    2557 scope.go:117] "RemoveContainer" containerID="f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525"
Jul 16 21:51:46.109653 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:51:46.109630    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525\": container with ID starting with f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525 not found: ID does not exist" containerID="f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525"
Jul 16 21:51:46.109719 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.109666    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525"} err="failed to get container status \"f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525\": rpc error: code = NotFound desc = could not find container \"f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525\": container with ID starting with f9e9136d6f7a0450af02d9016d082366ee40406e7d9b99e8a8596cd64f27c525 not found: ID does not exist"
Jul 16 21:51:46.139644 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.139620    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"]
Jul 16 21:51:46.156921 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.156899    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cd5bf47b9-b6s8g"]
Jul 16 21:51:46.173340 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173322    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-bound-sa-token\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173340 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173339    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a79de3dd-836d-4018-9482-974abea12707-ca-trust-extracted\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173349    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-registry-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173362    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-trusted-ca\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173374    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-installation-pull-secrets\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173390    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxm25\" (UniqueName: \"kubernetes.io/projected/a79de3dd-836d-4018-9482-974abea12707-kube-api-access-wxm25\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173398    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a79de3dd-836d-4018-9482-974abea12707-registry-certificates\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:46.173445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:46.173407    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a79de3dd-836d-4018-9482-974abea12707-image-registry-private-configuration\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:51:47.561653 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:47.561621    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79de3dd-836d-4018-9482-974abea12707" path="/var/lib/kubelet/pods/a79de3dd-836d-4018-9482-974abea12707/volumes"
Jul 16 21:51:56.139428 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:56.139397    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb11657a-5fd6-4f2d-8c12-fefc715c6bab" containerID="24a6635c08c42b31ca68eb831e1d77b155bb9b1922470a276a366e19a5bf1b17" exitCode=0
Jul 16 21:51:56.139763 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:56.139444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6" event={"ID":"fb11657a-5fd6-4f2d-8c12-fefc715c6bab","Type":"ContainerDied","Data":"24a6635c08c42b31ca68eb831e1d77b155bb9b1922470a276a366e19a5bf1b17"}
Jul 16 21:51:56.139763 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:56.139706    2557 scope.go:117] "RemoveContainer" containerID="24a6635c08c42b31ca68eb831e1d77b155bb9b1922470a276a366e19a5bf1b17"
Jul 16 21:51:57.145277 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:51:57.145242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-p2tm6" event={"ID":"fb11657a-5fd6-4f2d-8c12-fefc715c6bab","Type":"ContainerStarted","Data":"85d88282599a317da0fa72683507eb39a5f4b567ff13505cac68a4d44ce9a604"}
Jul 16 21:52:02.169911 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:02.169877    2557 generic.go:356] "Generic (PLEG): container finished" podID="3afdced7-a64f-4cac-858a-362b32874bd5" containerID="51f12f89a71e1342a1cf861e3bd89e5ffbbfd330e8f4ce6e67452aab49b86ea6" exitCode=0
Jul 16 21:52:02.170299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:02.169932    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-t4lc2" event={"ID":"3afdced7-a64f-4cac-858a-362b32874bd5","Type":"ContainerDied","Data":"51f12f89a71e1342a1cf861e3bd89e5ffbbfd330e8f4ce6e67452aab49b86ea6"}
Jul 16 21:52:02.170299 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:02.170211    2557 scope.go:117] "RemoveContainer" containerID="51f12f89a71e1342a1cf861e3bd89e5ffbbfd330e8f4ce6e67452aab49b86ea6"
Jul 16 21:52:03.175099 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:03.175066    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-t4lc2" event={"ID":"3afdced7-a64f-4cac-858a-362b32874bd5","Type":"ContainerStarted","Data":"d46bff784ee367ba7492b9493d395933f5020d46461b19b71dddc6e28be2f89e"}
Jul 16 21:52:06.187034 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:06.187001    2557 generic.go:356] "Generic (PLEG): container finished" podID="8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c" containerID="becf091a50091f98c6186dceef866309d30fa5dbb4af030bda6c8f5cf040cef6" exitCode=0
Jul 16 21:52:06.187372 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:06.187060    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf" event={"ID":"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c","Type":"ContainerDied","Data":"becf091a50091f98c6186dceef866309d30fa5dbb4af030bda6c8f5cf040cef6"}
Jul 16 21:52:06.187372 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:06.187298    2557 scope.go:117] "RemoveContainer" containerID="becf091a50091f98c6186dceef866309d30fa5dbb4af030bda6c8f5cf040cef6"
Jul 16 21:52:07.192732 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:07.192699    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-vtvvf" event={"ID":"8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c","Type":"ContainerStarted","Data":"b78cb8fb7277cdc3cb74ee901773d298807d84279c726040465c70ff41e8017e"}
Jul 16 21:52:51.352281 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:51.352254    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 21:52:51.352655 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:51.352293    2557 generic.go:356] "Generic (PLEG): container finished" podID="37dae182-a8ff-48b8-898e-155bda8bf0c5" containerID="f64c08e856015b86c76bb3b3b53fedeab2f2f1a69b2da8ccdd6c426a2775c072" exitCode=2
Jul 16 21:52:51.352655 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:51.352342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq" event={"ID":"37dae182-a8ff-48b8-898e-155bda8bf0c5","Type":"ContainerDied","Data":"f64c08e856015b86c76bb3b3b53fedeab2f2f1a69b2da8ccdd6c426a2775c072"}
Jul 16 21:52:51.352655 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:51.352589    2557 scope.go:117] "RemoveContainer" containerID="f64c08e856015b86c76bb3b3b53fedeab2f2f1a69b2da8ccdd6c426a2775c072"
Jul 16 21:52:52.357949 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:52.357924    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 21:52:52.358280 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:52:52.358040    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-nkrfq" event={"ID":"37dae182-a8ff-48b8-898e-155bda8bf0c5","Type":"ContainerStarted","Data":"ee863e17f2a7e9b91faf97c6a0627d1feadfd4a2d2961069e72f47cbf7009e61"}
Jul 16 21:53:45.125335 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.125304    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"]
Jul 16 21:53:45.125755 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.125591    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79de3dd-836d-4018-9482-974abea12707" containerName="registry"
Jul 16 21:53:45.125755 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.125602    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79de3dd-836d-4018-9482-974abea12707" containerName="registry"
Jul 16 21:53:45.125755 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.125668    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79de3dd-836d-4018-9482-974abea12707" containerName="registry"
Jul 16 21:53:45.128760 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.128745    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.131748 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.131727    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:53:45.131895 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.131877    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:53:45.131955 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.131883    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4tdm\""
Jul 16 21:53:45.136795 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.136771    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"]
Jul 16 21:53:45.286549 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.286517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.286723 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.286574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.286723 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.286654    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkvs\" (UniqueName: \"kubernetes.io/projected/43efe916-b8e4-43f8-a48c-b6454cd246d8-kube-api-access-dbkvs\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.387643 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.387550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.387643 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.387605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.387643 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.387643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkvs\" (UniqueName: \"kubernetes.io/projected/43efe916-b8e4-43f8-a48c-b6454cd246d8-kube-api-access-dbkvs\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.388004 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.387983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.388075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.388060    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.397254 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.397221    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkvs\" (UniqueName: \"kubernetes.io/projected/43efe916-b8e4-43f8-a48c-b6454cd246d8-kube-api-access-dbkvs\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.438235 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.438186    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:53:45.577137 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:45.577107    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"]
Jul 16 21:53:45.579750 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:53:45.579723    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43efe916_b8e4_43f8_a48c_b6454cd246d8.slice/crio-697365d20646ac92834c93502d43b5fdbbf8eeb51ab1dd101b0922769046c974 WatchSource:0}: Error finding container 697365d20646ac92834c93502d43b5fdbbf8eeb51ab1dd101b0922769046c974: Status 404 returned error can't find the container with id 697365d20646ac92834c93502d43b5fdbbf8eeb51ab1dd101b0922769046c974
Jul 16 21:53:46.557300 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:46.557259    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm" event={"ID":"43efe916-b8e4-43f8-a48c-b6454cd246d8","Type":"ContainerStarted","Data":"697365d20646ac92834c93502d43b5fdbbf8eeb51ab1dd101b0922769046c974"}
Jul 16 21:53:52.582222 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:52.582181    2557 generic.go:356] "Generic (PLEG): container finished" podID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerID="b02056701e00a293b77c7ffaa63f1af979425177d8912f828a7ec17c095f6ea6" exitCode=0
Jul 16 21:53:52.582711 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:52.582269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm" event={"ID":"43efe916-b8e4-43f8-a48c-b6454cd246d8","Type":"ContainerDied","Data":"b02056701e00a293b77c7ffaa63f1af979425177d8912f828a7ec17c095f6ea6"}
Jul 16 21:53:55.595591 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:55.595553    2557 generic.go:356] "Generic (PLEG): container finished" podID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerID="c6ac483565a20179299669b36a07036685d92e14a0d857483b4988f3f0d35375" exitCode=0
Jul 16 21:53:55.595986 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:53:55.595599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm" event={"ID":"43efe916-b8e4-43f8-a48c-b6454cd246d8","Type":"ContainerDied","Data":"c6ac483565a20179299669b36a07036685d92e14a0d857483b4988f3f0d35375"}
Jul 16 21:54:02.627724 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:02.627606    2557 generic.go:356] "Generic (PLEG): container finished" podID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerID="7dd0149b06f24658c120ff2fa2b15ff575c190bf83ae6ac6c5df6459dd6ec654" exitCode=0
Jul 16 21:54:02.627724 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:02.627710    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm" event={"ID":"43efe916-b8e4-43f8-a48c-b6454cd246d8","Type":"ContainerDied","Data":"7dd0149b06f24658c120ff2fa2b15ff575c190bf83ae6ac6c5df6459dd6ec654"}
Jul 16 21:54:03.753657 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.753631    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:54:03.851165 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.851130    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-bundle\") pod \"43efe916-b8e4-43f8-a48c-b6454cd246d8\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") "
Jul 16 21:54:03.851331 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.851236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-util\") pod \"43efe916-b8e4-43f8-a48c-b6454cd246d8\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") "
Jul 16 21:54:03.851331 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.851283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbkvs\" (UniqueName: \"kubernetes.io/projected/43efe916-b8e4-43f8-a48c-b6454cd246d8-kube-api-access-dbkvs\") pod \"43efe916-b8e4-43f8-a48c-b6454cd246d8\" (UID: \"43efe916-b8e4-43f8-a48c-b6454cd246d8\") "
Jul 16 21:54:03.851796 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.851765    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-bundle" (OuterVolumeSpecName: "bundle") pod "43efe916-b8e4-43f8-a48c-b6454cd246d8" (UID: "43efe916-b8e4-43f8-a48c-b6454cd246d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:54:03.853763 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.853742    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43efe916-b8e4-43f8-a48c-b6454cd246d8-kube-api-access-dbkvs" (OuterVolumeSpecName: "kube-api-access-dbkvs") pod "43efe916-b8e4-43f8-a48c-b6454cd246d8" (UID: "43efe916-b8e4-43f8-a48c-b6454cd246d8"). InnerVolumeSpecName "kube-api-access-dbkvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:54:03.855440 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.855416    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-util" (OuterVolumeSpecName: "util") pod "43efe916-b8e4-43f8-a48c-b6454cd246d8" (UID: "43efe916-b8e4-43f8-a48c-b6454cd246d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:54:03.951942 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.951905    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:54:03.951942 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.951942    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkvs\" (UniqueName: \"kubernetes.io/projected/43efe916-b8e4-43f8-a48c-b6454cd246d8-kube-api-access-dbkvs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:54:03.951942 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:03.951952    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43efe916-b8e4-43f8-a48c-b6454cd246d8-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:54:04.636835 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:04.636785    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm"
Jul 16 21:54:04.637009 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:04.636781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5f47mm" event={"ID":"43efe916-b8e4-43f8-a48c-b6454cd246d8","Type":"ContainerDied","Data":"697365d20646ac92834c93502d43b5fdbbf8eeb51ab1dd101b0922769046c974"}
Jul 16 21:54:04.637009 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:04.636930    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697365d20646ac92834c93502d43b5fdbbf8eeb51ab1dd101b0922769046c974"
Jul 16 21:54:12.581997 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.581908    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-gb26z"]
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582306    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="util"
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582320    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="util"
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582331    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="extract"
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582336    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="extract"
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582352    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="pull"
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582357    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="pull"
Jul 16 21:54:12.582544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.582406    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="43efe916-b8e4-43f8-a48c-b6454cd246d8" containerName="extract"
Jul 16 21:54:12.585301 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.585285    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.588038 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.588016    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 21:54:12.588169 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.588015    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 16 21:54:12.589011 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.588993    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 21:54:12.589105 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.589007    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 16 21:54:12.589105 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.589039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-79k9b\""
Jul 16 21:54:12.589105 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.589085    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 21:54:12.598613 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.598587    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-gb26z"]
Jul 16 21:54:12.623414 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.623380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.623414 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.623421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-cabundle0\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.623648 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.623439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfn9v\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-kube-api-access-vfn9v\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.724050 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.724013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.724050 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.724051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-cabundle0\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.724309 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.724070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfn9v\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-kube-api-access-vfn9v\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.724309 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:12.724166    2557 secret.go:281] references non-existent secret key: ca.crt
Jul 16 21:54:12.724309 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:12.724191    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 21:54:12.724309 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:12.724200    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-gb26z: references non-existent secret key: ca.crt
Jul 16 21:54:12.724309 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:12.724263    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates podName:aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a nodeName:}" failed. No retries permitted until 2026-07-16 21:54:13.224244612 +0000 UTC m=+264.203593891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates") pod "keda-operator-77f648d694-gb26z" (UID: "aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a") : references non-existent secret key: ca.crt
Jul 16 21:54:12.724761 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.724739    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-cabundle0\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.733587 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.733564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfn9v\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-kube-api-access-vfn9v\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:12.811293 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.811250    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"]
Jul 16 21:54:12.814561 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.814536    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:12.819171 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.819149    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 16 21:54:12.826848 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.826796    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"]
Jul 16 21:54:12.925154 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.925113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:12.925154 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.925154    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8bc13948-57d8-4b09-9519-ecb443c0435f-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:12.925382 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:12.925261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvfvg\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-kube-api-access-mvfvg\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.026626 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.026584    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.026626 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.026628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8bc13948-57d8-4b09-9519-ecb443c0435f-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.026921 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.026680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvfvg\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-kube-api-access-mvfvg\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.026921 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.026732    2557 secret.go:281] references non-existent secret key: tls.crt
Jul 16 21:54:13.026921 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.026752    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 16 21:54:13.026921 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.026770    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr: references non-existent secret key: tls.crt
Jul 16 21:54:13.026921 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.026861    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates podName:8bc13948-57d8-4b09-9519-ecb443c0435f nodeName:}" failed. No retries permitted until 2026-07-16 21:54:13.526842278 +0000 UTC m=+264.506191579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates") pod "keda-metrics-apiserver-55557df8c-tsrxr" (UID: "8bc13948-57d8-4b09-9519-ecb443c0435f") : references non-existent secret key: tls.crt
Jul 16 21:54:13.027158 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.027057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8bc13948-57d8-4b09-9519-ecb443c0435f-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.041189 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.041147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvfvg\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-kube-api-access-mvfvg\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.228070 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.227971    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:13.228223 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.228134    2557 secret.go:281] references non-existent secret key: ca.crt
Jul 16 21:54:13.228223 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.228153    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 21:54:13.228223 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.228162    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-gb26z: references non-existent secret key: ca.crt
Jul 16 21:54:13.228324 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.228227    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates podName:aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a nodeName:}" failed. No retries permitted until 2026-07-16 21:54:14.228211299 +0000 UTC m=+265.207560576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates") pod "keda-operator-77f648d694-gb26z" (UID: "aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a") : references non-existent secret key: ca.crt
Jul 16 21:54:13.529946 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.529846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:13.530125 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.530007    2557 secret.go:281] references non-existent secret key: tls.crt
Jul 16 21:54:13.530125 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.530031    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 16 21:54:13.530125 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.530051    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr: references non-existent secret key: tls.crt
Jul 16 21:54:13.530125 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:54:13.530119    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates podName:8bc13948-57d8-4b09-9519-ecb443c0435f nodeName:}" failed. No retries permitted until 2026-07-16 21:54:14.530100641 +0000 UTC m=+265.509449918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates") pod "keda-metrics-apiserver-55557df8c-tsrxr" (UID: "8bc13948-57d8-4b09-9519-ecb443c0435f") : references non-existent secret key: tls.crt
Jul 16 21:54:13.640205 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.640173    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-dbc2x"]
Jul 16 21:54:13.643421 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.643406    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.646445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.646423    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 16 21:54:13.666848 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.666803    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-dbc2x"]
Jul 16 21:54:13.731916 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.731884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqv4x\" (UniqueName: \"kubernetes.io/projected/c8b76cf4-2c7e-438c-88ea-d6e66178ec7b-kube-api-access-wqv4x\") pod \"keda-admission-56f9798bb4-dbc2x\" (UID: \"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b\") " pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.732101 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.731931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c8b76cf4-2c7e-438c-88ea-d6e66178ec7b-certificates\") pod \"keda-admission-56f9798bb4-dbc2x\" (UID: \"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b\") " pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.833447 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.833354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqv4x\" (UniqueName: \"kubernetes.io/projected/c8b76cf4-2c7e-438c-88ea-d6e66178ec7b-kube-api-access-wqv4x\") pod \"keda-admission-56f9798bb4-dbc2x\" (UID: \"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b\") " pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.833447 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.833398    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c8b76cf4-2c7e-438c-88ea-d6e66178ec7b-certificates\") pod \"keda-admission-56f9798bb4-dbc2x\" (UID: \"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b\") " pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.836156 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.836132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c8b76cf4-2c7e-438c-88ea-d6e66178ec7b-certificates\") pod \"keda-admission-56f9798bb4-dbc2x\" (UID: \"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b\") " pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.858033 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.858007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqv4x\" (UniqueName: \"kubernetes.io/projected/c8b76cf4-2c7e-438c-88ea-d6e66178ec7b-kube-api-access-wqv4x\") pod \"keda-admission-56f9798bb4-dbc2x\" (UID: \"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b\") " pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:13.953197 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:13.953160    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:14.091544 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.091397    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-dbc2x"]
Jul 16 21:54:14.094382 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:54:14.094356    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b76cf4_2c7e_438c_88ea_d6e66178ec7b.slice/crio-3f914923b9411f132fe93850f18c0645fc8f701adbc406e97a1cf1f4e314d7c3 WatchSource:0}: Error finding container 3f914923b9411f132fe93850f18c0645fc8f701adbc406e97a1cf1f4e314d7c3: Status 404 returned error can't find the container with id 3f914923b9411f132fe93850f18c0645fc8f701adbc406e97a1cf1f4e314d7c3
Jul 16 21:54:14.236456 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.236422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:14.238990 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.238963    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a-certificates\") pod \"keda-operator-77f648d694-gb26z\" (UID: \"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a\") " pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:14.395180 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.395153    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:14.531183 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.531149    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-gb26z"]
Jul 16 21:54:14.534301 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:54:14.534280    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa317d5c_ba9b_4e5d_9be4_d5429eb4ef1a.slice/crio-03004e9792395b657bb3769dd15bdbfa0a27735de839fabaa0b162b9cd6baee2 WatchSource:0}: Error finding container 03004e9792395b657bb3769dd15bdbfa0a27735de839fabaa0b162b9cd6baee2: Status 404 returned error can't find the container with id 03004e9792395b657bb3769dd15bdbfa0a27735de839fabaa0b162b9cd6baee2
Jul 16 21:54:14.538538 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.538518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:14.540999 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.540977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc13948-57d8-4b09-9519-ecb443c0435f-certificates\") pod \"keda-metrics-apiserver-55557df8c-tsrxr\" (UID: \"8bc13948-57d8-4b09-9519-ecb443c0435f\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:14.624602 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.624577    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:14.680719 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.680690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-gb26z" event={"ID":"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a","Type":"ContainerStarted","Data":"03004e9792395b657bb3769dd15bdbfa0a27735de839fabaa0b162b9cd6baee2"}
Jul 16 21:54:14.682091 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.682062    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x" event={"ID":"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b","Type":"ContainerStarted","Data":"3f914923b9411f132fe93850f18c0645fc8f701adbc406e97a1cf1f4e314d7c3"}
Jul 16 21:54:14.762651 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:14.762623    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"]
Jul 16 21:54:14.765324 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:54:14.765291    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc13948_57d8_4b09_9519_ecb443c0435f.slice/crio-2c23ba5c63bbdce991be0339987082c37a02fb409c3c02139ef5fe278c7b37e9 WatchSource:0}: Error finding container 2c23ba5c63bbdce991be0339987082c37a02fb409c3c02139ef5fe278c7b37e9: Status 404 returned error can't find the container with id 2c23ba5c63bbdce991be0339987082c37a02fb409c3c02139ef5fe278c7b37e9
Jul 16 21:54:15.688669 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:15.688605    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr" event={"ID":"8bc13948-57d8-4b09-9519-ecb443c0435f","Type":"ContainerStarted","Data":"2c23ba5c63bbdce991be0339987082c37a02fb409c3c02139ef5fe278c7b37e9"}
Jul 16 21:54:16.695479 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:16.695423    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x" event={"ID":"c8b76cf4-2c7e-438c-88ea-d6e66178ec7b","Type":"ContainerStarted","Data":"d89ac8daafef757bc09145036935c3cae47bb8dfd88be836fc20a80fc505aea4"}
Jul 16 21:54:16.695921 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:16.695526    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:16.712421 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:16.712364    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x" podStartSLOduration=2.138522499 podStartE2EDuration="3.712344786s" totalImagesPullingTime="1.573822287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:16.711778512 +0000 UTC m=+267.691127866" watchObservedRunningTime="2026-07-16 21:54:16.712344786 +0000 UTC m=+267.691694086"
Jul 16 21:54:19.709476 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:19.709441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr" event={"ID":"8bc13948-57d8-4b09-9519-ecb443c0435f","Type":"ContainerStarted","Data":"3e2487b194b89a0308dd8c63db31352f3370d01154cefd37823da38f035af10c"}
Jul 16 21:54:19.709951 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:19.709594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:19.710914 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:19.710894    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-gb26z" event={"ID":"aa317d5c-ba9b-4e5d-9be4-d5429eb4ef1a","Type":"ContainerStarted","Data":"acf33bcf179c554cc5e7e52017e96b8115c1bd8f32b73fb7197f9ed98a1095d3"}
Jul 16 21:54:19.711032 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:19.711019    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:19.731582 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:19.731537    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr" podStartSLOduration=3.832068698 podStartE2EDuration="7.731522824s" totalImagesPullingTime="3.899454126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:19.72964847 +0000 UTC m=+270.708997768" watchObservedRunningTime="2026-07-16 21:54:19.731522824 +0000 UTC m=+270.710872123"
Jul 16 21:54:19.749547 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:19.749500    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-gb26z" podStartSLOduration=3.619538263 podStartE2EDuration="7.749487861s" totalImagesPullingTime="4.129949598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:19.74928623 +0000 UTC m=+270.728635530" watchObservedRunningTime="2026-07-16 21:54:19.749487861 +0000 UTC m=+270.728837160"
Jul 16 21:54:30.719099 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:30.719069    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tsrxr"
Jul 16 21:54:37.703897 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:37.703862    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-dbc2x"
Jul 16 21:54:40.717740 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:40.717708    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-gb26z"
Jul 16 21:54:49.421781 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:49.421753    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 21:54:49.426359 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:49.426337    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 21:54:49.428229 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:49.428204    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 21:54:49.435836 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:49.433366    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 21:54:49.444908 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:54:49.444887    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:55:21.175668 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.175630    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-5hk5f"]
Jul 16 21:55:21.177894 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.177879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.181135 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.181101    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 16 21:55:21.181252 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.181138    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 21:55:21.181252 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.181108    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 21:55:21.181962 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.181943    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vdpjb\""
Jul 16 21:55:21.189334 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.189314    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-5hk5f"]
Jul 16 21:55:21.226660 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.226630    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-gdzxl"]
Jul 16 21:55:21.229060 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.229043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.231274 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.231249    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v4cv8\""
Jul 16 21:55:21.231387 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.231332    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 16 21:55:21.238634 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.238613    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gdzxl"]
Jul 16 21:55:21.303616 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.303579    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfhm\" (UniqueName: \"kubernetes.io/projected/64809f09-8938-4fcb-88ac-64f0ae3cf640-kube-api-access-phfhm\") pod \"kserve-controller-manager-84cf6d888d-5hk5f\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") " pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.303798 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.303633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cb76a0a-af3e-4721-9a0a-dabca6f53dd7-data\") pod \"seaweedfs-64568bcd49-gdzxl\" (UID: \"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7\") " pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.303798 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.303694    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwgk\" (UniqueName: \"kubernetes.io/projected/5cb76a0a-af3e-4721-9a0a-dabca6f53dd7-kube-api-access-5jwgk\") pod \"seaweedfs-64568bcd49-gdzxl\" (UID: \"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7\") " pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.303798 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.303717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64809f09-8938-4fcb-88ac-64f0ae3cf640-cert\") pod \"kserve-controller-manager-84cf6d888d-5hk5f\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") " pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.405058 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.405018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cb76a0a-af3e-4721-9a0a-dabca6f53dd7-data\") pod \"seaweedfs-64568bcd49-gdzxl\" (UID: \"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7\") " pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.405261 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.405076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwgk\" (UniqueName: \"kubernetes.io/projected/5cb76a0a-af3e-4721-9a0a-dabca6f53dd7-kube-api-access-5jwgk\") pod \"seaweedfs-64568bcd49-gdzxl\" (UID: \"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7\") " pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.405261 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.405107    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64809f09-8938-4fcb-88ac-64f0ae3cf640-cert\") pod \"kserve-controller-manager-84cf6d888d-5hk5f\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") " pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.405261 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.405178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfhm\" (UniqueName: \"kubernetes.io/projected/64809f09-8938-4fcb-88ac-64f0ae3cf640-kube-api-access-phfhm\") pod \"kserve-controller-manager-84cf6d888d-5hk5f\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") " pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.405505 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.405480    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cb76a0a-af3e-4721-9a0a-dabca6f53dd7-data\") pod \"seaweedfs-64568bcd49-gdzxl\" (UID: \"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7\") " pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.407751 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.407729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64809f09-8938-4fcb-88ac-64f0ae3cf640-cert\") pod \"kserve-controller-manager-84cf6d888d-5hk5f\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") " pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.421998 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.421972    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwgk\" (UniqueName: \"kubernetes.io/projected/5cb76a0a-af3e-4721-9a0a-dabca6f53dd7-kube-api-access-5jwgk\") pod \"seaweedfs-64568bcd49-gdzxl\" (UID: \"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7\") " pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.432236 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.432178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfhm\" (UniqueName: \"kubernetes.io/projected/64809f09-8938-4fcb-88ac-64f0ae3cf640-kube-api-access-phfhm\") pod \"kserve-controller-manager-84cf6d888d-5hk5f\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") " pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.488167 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.488135    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:21.539599 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.539563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:21.657891 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.657680    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-5hk5f"]
Jul 16 21:55:21.662408 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:55:21.662378    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64809f09_8938_4fcb_88ac_64f0ae3cf640.slice/crio-785e78dc93799cda0e2c81920fa30a7d58fbabf77ca7061d209e2044c9dd9bc6 WatchSource:0}: Error finding container 785e78dc93799cda0e2c81920fa30a7d58fbabf77ca7061d209e2044c9dd9bc6: Status 404 returned error can't find the container with id 785e78dc93799cda0e2c81920fa30a7d58fbabf77ca7061d209e2044c9dd9bc6
Jul 16 21:55:21.663766 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.663745    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:55:21.709670 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.709648    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gdzxl"]
Jul 16 21:55:21.712052 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:55:21.712027    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb76a0a_af3e_4721_9a0a_dabca6f53dd7.slice/crio-89b2cb3954dd99a12f5ba9fd07568f9876aed532c2ccba16b3030a28dafda7ca WatchSource:0}: Error finding container 89b2cb3954dd99a12f5ba9fd07568f9876aed532c2ccba16b3030a28dafda7ca: Status 404 returned error can't find the container with id 89b2cb3954dd99a12f5ba9fd07568f9876aed532c2ccba16b3030a28dafda7ca
Jul 16 21:55:21.958211 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.958179    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gdzxl" event={"ID":"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7","Type":"ContainerStarted","Data":"89b2cb3954dd99a12f5ba9fd07568f9876aed532c2ccba16b3030a28dafda7ca"}
Jul 16 21:55:21.959376 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:21.959348    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f" event={"ID":"64809f09-8938-4fcb-88ac-64f0ae3cf640","Type":"ContainerStarted","Data":"785e78dc93799cda0e2c81920fa30a7d58fbabf77ca7061d209e2044c9dd9bc6"}
Jul 16 21:55:25.981053 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:25.981017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gdzxl" event={"ID":"5cb76a0a-af3e-4721-9a0a-dabca6f53dd7","Type":"ContainerStarted","Data":"f916cf15475636c1d1e3a02f1f7c71585fb2bc557dc069fd148fc7c5cd1c56b8"}
Jul 16 21:55:25.981534 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:25.981156    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:25.982493 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:25.982463    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f" event={"ID":"64809f09-8938-4fcb-88ac-64f0ae3cf640","Type":"ContainerStarted","Data":"d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541"}
Jul 16 21:55:25.982610 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:25.982588    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:55:26.001464 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:26.001415    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-gdzxl" podStartSLOduration=0.979851947 podStartE2EDuration="5.001400438s" totalImagesPullingTime="4.021548491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:25.99964729 +0000 UTC m=+336.978996588" watchObservedRunningTime="2026-07-16 21:55:26.001400438 +0000 UTC m=+336.980749737"
Jul 16 21:55:26.029418 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:26.029364    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f" podStartSLOduration=0.994106813 podStartE2EDuration="5.029346583s" totalImagesPullingTime="4.03523977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:26.02929274 +0000 UTC m=+337.008642041" watchObservedRunningTime="2026-07-16 21:55:26.029346583 +0000 UTC m=+337.008695884"
Jul 16 21:55:31.988827 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:31.988793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-gdzxl"
Jul 16 21:55:56.993265 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:55:56.993227    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:56:04.990070 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:04.990017    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"]
Jul 16 21:56:04.997378 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:04.997356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.007020 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.006995    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"]
Jul 16 21:56:05.010200 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.010184    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:56:05.010411 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.010395    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:56:05.010726 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.010710    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4tdm\""
Jul 16 21:56:05.078939 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.078901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.078939 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.078942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srw2g\" (UniqueName: \"kubernetes.io/projected/7a44f073-b3c3-484f-959a-911667ff5f28-kube-api-access-srw2g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.079183 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.079054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.179930 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.179896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.180138 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.179949    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.180138 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.179968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srw2g\" (UniqueName: \"kubernetes.io/projected/7a44f073-b3c3-484f-959a-911667ff5f28-kube-api-access-srw2g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.180361 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.180335    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.180427 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.180366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.189922 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.189899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srw2g\" (UniqueName: \"kubernetes.io/projected/7a44f073-b3c3-484f-959a-911667ff5f28-kube-api-access-srw2g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.307108 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.307011    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:05.456981 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:05.456953    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"]
Jul 16 21:56:05.458738 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:56:05.458713    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a44f073_b3c3_484f_959a_911667ff5f28.slice/crio-c3e9d82a315e40e9161674b4512323d0f8fb95d5f489c9cffb346e13d302e626 WatchSource:0}: Error finding container c3e9d82a315e40e9161674b4512323d0f8fb95d5f489c9cffb346e13d302e626: Status 404 returned error can't find the container with id c3e9d82a315e40e9161674b4512323d0f8fb95d5f489c9cffb346e13d302e626
Jul 16 21:56:06.150079 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:06.150038    2557 generic.go:356] "Generic (PLEG): container finished" podID="7a44f073-b3c3-484f-959a-911667ff5f28" containerID="aaa20e05e7654a0a53bce97953657aa5a4b9e7077782ba3e10dce0dddfae904e" exitCode=0
Jul 16 21:56:06.150476 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:06.150103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64" event={"ID":"7a44f073-b3c3-484f-959a-911667ff5f28","Type":"ContainerDied","Data":"aaa20e05e7654a0a53bce97953657aa5a4b9e7077782ba3e10dce0dddfae904e"}
Jul 16 21:56:06.150476 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:06.150156    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64" event={"ID":"7a44f073-b3c3-484f-959a-911667ff5f28","Type":"ContainerStarted","Data":"c3e9d82a315e40e9161674b4512323d0f8fb95d5f489c9cffb346e13d302e626"}
Jul 16 21:56:09.171649 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:09.171615    2557 generic.go:356] "Generic (PLEG): container finished" podID="7a44f073-b3c3-484f-959a-911667ff5f28" containerID="e31e0f83c90c265911d0421251b73c03da91c289972eed93a1c05ad5ae9e90b3" exitCode=0
Jul 16 21:56:09.172066 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:09.171673    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64" event={"ID":"7a44f073-b3c3-484f-959a-911667ff5f28","Type":"ContainerDied","Data":"e31e0f83c90c265911d0421251b73c03da91c289972eed93a1c05ad5ae9e90b3"}
Jul 16 21:56:10.179145 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:10.179103    2557 generic.go:356] "Generic (PLEG): container finished" podID="7a44f073-b3c3-484f-959a-911667ff5f28" containerID="606e24d912a094ca75cdef19abf3f1f64eb61e26ff892103664e7fcf190373d8" exitCode=0
Jul 16 21:56:10.179547 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:10.179180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64" event={"ID":"7a44f073-b3c3-484f-959a-911667ff5f28","Type":"ContainerDied","Data":"606e24d912a094ca75cdef19abf3f1f64eb61e26ff892103664e7fcf190373d8"}
Jul 16 21:56:11.310230 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.310204    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:11.431725 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.431690    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-util\") pod \"7a44f073-b3c3-484f-959a-911667ff5f28\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") "
Jul 16 21:56:11.431951 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.431765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-bundle\") pod \"7a44f073-b3c3-484f-959a-911667ff5f28\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") "
Jul 16 21:56:11.431951 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.431802    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srw2g\" (UniqueName: \"kubernetes.io/projected/7a44f073-b3c3-484f-959a-911667ff5f28-kube-api-access-srw2g\") pod \"7a44f073-b3c3-484f-959a-911667ff5f28\" (UID: \"7a44f073-b3c3-484f-959a-911667ff5f28\") "
Jul 16 21:56:11.432517 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.432493    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-bundle" (OuterVolumeSpecName: "bundle") pod "7a44f073-b3c3-484f-959a-911667ff5f28" (UID: "7a44f073-b3c3-484f-959a-911667ff5f28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:56:11.434074 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.434014    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a44f073-b3c3-484f-959a-911667ff5f28-kube-api-access-srw2g" (OuterVolumeSpecName: "kube-api-access-srw2g") pod "7a44f073-b3c3-484f-959a-911667ff5f28" (UID: "7a44f073-b3c3-484f-959a-911667ff5f28"). InnerVolumeSpecName "kube-api-access-srw2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:11.436659 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.436633    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-util" (OuterVolumeSpecName: "util") pod "7a44f073-b3c3-484f-959a-911667ff5f28" (UID: "7a44f073-b3c3-484f-959a-911667ff5f28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:56:11.533294 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.533257    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:56:11.533294 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.533286    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a44f073-b3c3-484f-959a-911667ff5f28-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:56:11.533294 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:11.533298    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srw2g\" (UniqueName: \"kubernetes.io/projected/7a44f073-b3c3-484f-959a-911667ff5f28-kube-api-access-srw2g\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:56:12.189657 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:12.189621    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64" event={"ID":"7a44f073-b3c3-484f-959a-911667ff5f28","Type":"ContainerDied","Data":"c3e9d82a315e40e9161674b4512323d0f8fb95d5f489c9cffb346e13d302e626"}
Jul 16 21:56:12.189657 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:12.189656    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x8c64"
Jul 16 21:56:12.189933 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:12.189673    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e9d82a315e40e9161674b4512323d0f8fb95d5f489c9cffb346e13d302e626"
Jul 16 21:56:27.075237 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075201    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"]
Jul 16 21:56:27.075633 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075547    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="extract"
Jul 16 21:56:27.075633 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075558    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="extract"
Jul 16 21:56:27.075633 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="util"
Jul 16 21:56:27.075633 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075574    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="util"
Jul 16 21:56:27.075633 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075582    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="pull"
Jul 16 21:56:27.075633 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075587    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="pull"
Jul 16 21:56:27.075878 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.075642    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a44f073-b3c3-484f-959a-911667ff5f28" containerName="extract"
Jul 16 21:56:27.079016 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.078999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.084325 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.084299    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:56:27.084502 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.084320    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:56:27.084648 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.084346    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4tdm\""
Jul 16 21:56:27.103856 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.103808    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"]
Jul 16 21:56:27.170605 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.170569    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24t72\" (UniqueName: \"kubernetes.io/projected/1ecae7ab-6309-4819-b7d6-455dfec060ea-kube-api-access-24t72\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.170778 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.170686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.170778 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.170723    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.271593 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.271560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.271763 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.271606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24t72\" (UniqueName: \"kubernetes.io/projected/1ecae7ab-6309-4819-b7d6-455dfec060ea-kube-api-access-24t72\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.271763 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.271666    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.271940 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.271922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.272003 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.271957    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.284760 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.284732    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24t72\" (UniqueName: \"kubernetes.io/projected/1ecae7ab-6309-4819-b7d6-455dfec060ea-kube-api-access-24t72\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.388140 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.388043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:27.533398 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:27.533367    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"]
Jul 16 21:56:27.536762 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:56:27.536734    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecae7ab_6309_4819_b7d6_455dfec060ea.slice/crio-647461bd401b22314d05db15eae7dcdde04d391e358272a665e2ce4869bc701b WatchSource:0}: Error finding container 647461bd401b22314d05db15eae7dcdde04d391e358272a665e2ce4869bc701b: Status 404 returned error can't find the container with id 647461bd401b22314d05db15eae7dcdde04d391e358272a665e2ce4869bc701b
Jul 16 21:56:28.264786 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:28.264751    2557 generic.go:356] "Generic (PLEG): container finished" podID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerID="f0a8e052b5dfdaaef71cd3d303d310a1190a1267ecddf9fe5f93829a485ca16b" exitCode=0
Jul 16 21:56:28.265399 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:28.264847    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2" event={"ID":"1ecae7ab-6309-4819-b7d6-455dfec060ea","Type":"ContainerDied","Data":"f0a8e052b5dfdaaef71cd3d303d310a1190a1267ecddf9fe5f93829a485ca16b"}
Jul 16 21:56:28.265399 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:28.264897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2" event={"ID":"1ecae7ab-6309-4819-b7d6-455dfec060ea","Type":"ContainerStarted","Data":"647461bd401b22314d05db15eae7dcdde04d391e358272a665e2ce4869bc701b"}
Jul 16 21:56:31.280145 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:31.280103    2557 generic.go:356] "Generic (PLEG): container finished" podID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerID="3e08c98a39b4a5b6e297d3e8fc32d7f0b7c03e4731078e0a7b93b722b7068515" exitCode=0
Jul 16 21:56:31.280532 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:31.280190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2" event={"ID":"1ecae7ab-6309-4819-b7d6-455dfec060ea","Type":"ContainerDied","Data":"3e08c98a39b4a5b6e297d3e8fc32d7f0b7c03e4731078e0a7b93b722b7068515"}
Jul 16 21:56:32.287171 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:32.287137    2557 generic.go:356] "Generic (PLEG): container finished" podID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerID="0eaf7df6bfc5e260912fd270b0cc4d0cf3832515134cd76664ef31a622e297ac" exitCode=0
Jul 16 21:56:32.287592 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:32.287233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2" event={"ID":"1ecae7ab-6309-4819-b7d6-455dfec060ea","Type":"ContainerDied","Data":"0eaf7df6bfc5e260912fd270b0cc4d0cf3832515134cd76664ef31a622e297ac"}
Jul 16 21:56:33.418683 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.418656    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:33.525591 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.525558    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-util\") pod \"1ecae7ab-6309-4819-b7d6-455dfec060ea\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") "
Jul 16 21:56:33.525782 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.525598    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-bundle\") pod \"1ecae7ab-6309-4819-b7d6-455dfec060ea\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") "
Jul 16 21:56:33.525782 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.525630    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24t72\" (UniqueName: \"kubernetes.io/projected/1ecae7ab-6309-4819-b7d6-455dfec060ea-kube-api-access-24t72\") pod \"1ecae7ab-6309-4819-b7d6-455dfec060ea\" (UID: \"1ecae7ab-6309-4819-b7d6-455dfec060ea\") "
Jul 16 21:56:33.526053 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.526031    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-bundle" (OuterVolumeSpecName: "bundle") pod "1ecae7ab-6309-4819-b7d6-455dfec060ea" (UID: "1ecae7ab-6309-4819-b7d6-455dfec060ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:56:33.527915 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.527889    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecae7ab-6309-4819-b7d6-455dfec060ea-kube-api-access-24t72" (OuterVolumeSpecName: "kube-api-access-24t72") pod "1ecae7ab-6309-4819-b7d6-455dfec060ea" (UID: "1ecae7ab-6309-4819-b7d6-455dfec060ea"). InnerVolumeSpecName "kube-api-access-24t72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:33.530586 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.530565    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-util" (OuterVolumeSpecName: "util") pod "1ecae7ab-6309-4819-b7d6-455dfec060ea" (UID: "1ecae7ab-6309-4819-b7d6-455dfec060ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:56:33.626703 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.626672    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24t72\" (UniqueName: \"kubernetes.io/projected/1ecae7ab-6309-4819-b7d6-455dfec060ea-kube-api-access-24t72\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:56:33.626703 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.626697    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:56:33.626703 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:33.626706    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ecae7ab-6309-4819-b7d6-455dfec060ea-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:56:34.297296 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:34.297265    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2"
Jul 16 21:56:34.297468 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:34.297298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzcr2" event={"ID":"1ecae7ab-6309-4819-b7d6-455dfec060ea","Type":"ContainerDied","Data":"647461bd401b22314d05db15eae7dcdde04d391e358272a665e2ce4869bc701b"}
Jul 16 21:56:34.297468 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:34.297340    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647461bd401b22314d05db15eae7dcdde04d391e358272a665e2ce4869bc701b"
Jul 16 21:56:56.775201 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775167    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"]
Jul 16 21:56:56.775571 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775489    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="pull"
Jul 16 21:56:56.775571 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775500    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="pull"
Jul 16 21:56:56.775571 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775508    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="extract"
Jul 16 21:56:56.775571 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775514    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="extract"
Jul 16 21:56:56.775571 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775531    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="util"
Jul 16 21:56:56.775571 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775536    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="util"
Jul 16 21:56:56.775754 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.775582    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecae7ab-6309-4819-b7d6-455dfec060ea" containerName="extract"
Jul 16 21:56:56.778592 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.778576    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.781282 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.781263    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 21:56:56.781480 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.781461    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f8s5f\""
Jul 16 21:56:56.781575 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.781555    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 21:56:56.781769 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.781755    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 21:56:56.781902 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.781887    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 21:56:56.902828 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.902783    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"]
Jul 16 21:56:56.917103 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.917225 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917123    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.917225 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917144    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.917225 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqn7\" (UniqueName: \"kubernetes.io/projected/e4700a47-8782-4770-be87-5f9c000ef5a1-kube-api-access-cgqn7\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.917225 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4700a47-8782-4770-be87-5f9c000ef5a1-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.917358 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:56.917358 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:56.917291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.018732 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.018694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.018732 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.018741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.018987 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.018772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.018987 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.018801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.019089 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.019010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.019089 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.019072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqn7\" (UniqueName: \"kubernetes.io/projected/e4700a47-8782-4770-be87-5f9c000ef5a1-kube-api-access-cgqn7\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.019169 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.019132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4700a47-8782-4770-be87-5f9c000ef5a1-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.019507 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.019476    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.021425 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.021396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.021522 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.021474    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4700a47-8782-4770-be87-5f9c000ef5a1-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.021560 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.021520    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.021560 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.021514    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4700a47-8782-4770-be87-5f9c000ef5a1-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.051778 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.051722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4700a47-8782-4770-be87-5f9c000ef5a1-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.051915 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.051897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqn7\" (UniqueName: \"kubernetes.io/projected/e4700a47-8782-4770-be87-5f9c000ef5a1-kube-api-access-cgqn7\") pod \"istiod-openshift-gateway-94bb8fbfd-wgrrr\" (UID: \"e4700a47-8782-4770-be87-5f9c000ef5a1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.087941 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.087917    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:56:57.254218 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.254194    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"]
Jul 16 21:56:57.255694 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:56:57.255667    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4700a47_8782_4770_be87_5f9c000ef5a1.slice/crio-f0047884a2992e39005768135aeae623730f448cb62a242a1c744ee687cafd7a WatchSource:0}: Error finding container f0047884a2992e39005768135aeae623730f448cb62a242a1c744ee687cafd7a: Status 404 returned error can't find the container with id f0047884a2992e39005768135aeae623730f448cb62a242a1c744ee687cafd7a
Jul 16 21:56:57.400034 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:57.400004    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr" event={"ID":"e4700a47-8782-4770-be87-5f9c000ef5a1","Type":"ContainerStarted","Data":"f0047884a2992e39005768135aeae623730f448cb62a242a1c744ee687cafd7a"}
Jul 16 21:56:59.468785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:59.468736    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:56:59.469102 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:56:59.468851    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:57:00.418841 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:00.418795    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr" event={"ID":"e4700a47-8782-4770-be87-5f9c000ef5a1","Type":"ContainerStarted","Data":"fed0195403e8c0137b40cd890f92c83dbb3618e1b43e9d12e1c2f7ed7fccfba5"}
Jul 16 21:57:00.419053 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:00.419035    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:57:00.420846 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:00.420798    2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-wgrrr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 21:57:00.420974 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:00.420874    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr" podUID="e4700a47-8782-4770-be87-5f9c000ef5a1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:57:01.425151 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:01.425113    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr"
Jul 16 21:57:01.475205 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:01.475150    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wgrrr" podStartSLOduration=3.26422939 podStartE2EDuration="5.475132962s" totalImagesPullingTime="2.210903572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:00.498952864 +0000 UTC m=+431.478302163" watchObservedRunningTime="2026-07-16 21:57:01.475132962 +0000 UTC m=+432.454482261"
Jul 16 21:57:14.689428 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.689345    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"]
Jul 16 21:57:14.692914 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.692897    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.697832 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.697796    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4tdm\""
Jul 16 21:57:14.698212 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.698192    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:57:14.698899 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.698878    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:57:14.707346 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.707323    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"]
Jul 16 21:57:14.770231 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.770199    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"]
Jul 16 21:57:14.773593 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.773579    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.775946 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.775923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.776065 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.775983    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.776065 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.776036    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8fz\" (UniqueName: \"kubernetes.io/projected/c40f2307-e006-4b84-a5bf-3784f32b595a-kube-api-access-fh8fz\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.793257 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.793231    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"]
Jul 16 21:57:14.853136 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.853101    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"]
Jul 16 21:57:14.856557 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.856538    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.872166 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.872143    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"]
Jul 16 21:57:14.876950 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.876928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngw8\" (UniqueName: \"kubernetes.io/projected/644d2f11-9d76-480d-9cc3-86a5de82cd18-kube-api-access-lngw8\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.877058 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.876987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.877058 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877016    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.877058 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877040    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.877230 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.877230 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877152    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.877230 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8fz\" (UniqueName: \"kubernetes.io/projected/c40f2307-e006-4b84-a5bf-3784f32b595a-kube-api-access-fh8fz\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.877370 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877235    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.877370 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877264    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9g4\" (UniqueName: \"kubernetes.io/projected/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-kube-api-access-tm9g4\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.877370 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877287    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.877513 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.877494    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.892720 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.892688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8fz\" (UniqueName: \"kubernetes.io/projected/c40f2307-e006-4b84-a5bf-3784f32b595a-kube-api-access-fh8fz\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:14.959258 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.959198    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"]
Jul 16 21:57:14.963021 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.963006    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:14.974708 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.974685    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"]
Jul 16 21:57:14.978223 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978200    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.978303 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978236    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9g4\" (UniqueName: \"kubernetes.io/projected/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-kube-api-access-tm9g4\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.978303 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngw8\" (UniqueName: \"kubernetes.io/projected/644d2f11-9d76-480d-9cc3-86a5de82cd18-kube-api-access-lngw8\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.978401 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzvk\" (UniqueName: \"kubernetes.io/projected/1721e48a-e57a-494c-b617-a5aa37fff968-kube-api-access-4vzvk\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:14.978433 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.978471 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978425    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.978511 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.978563 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:14.978618 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:14.978618 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978581    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.978799 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978780    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:14.978875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:14.978875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:14.978840    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:15.000562 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.000538    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngw8\" (UniqueName: \"kubernetes.io/projected/644d2f11-9d76-480d-9cc3-86a5de82cd18-kube-api-access-lngw8\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:15.000646 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.000611    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9g4\" (UniqueName: \"kubernetes.io/projected/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-kube-api-access-tm9g4\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:15.002424 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.002406    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:15.079793 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.079747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.079971 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.079853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzvk\" (UniqueName: \"kubernetes.io/projected/1721e48a-e57a-494c-b617-a5aa37fff968-kube-api-access-4vzvk\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.079971 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.079900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.080220 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.080197    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.080284 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.080222    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.082792 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.082633    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:15.092162 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.092138    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzvk\" (UniqueName: \"kubernetes.io/projected/1721e48a-e57a-494c-b617-a5aa37fff968-kube-api-access-4vzvk\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.151253 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.151222    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"]
Jul 16 21:57:15.154692 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:15.154621    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644d2f11_9d76_480d_9cc3_86a5de82cd18.slice/crio-2b0620f2b120d71bb09a4b146b409c1a58fceadcd60e7ff0017969199bab2042 WatchSource:0}: Error finding container 2b0620f2b120d71bb09a4b146b409c1a58fceadcd60e7ff0017969199bab2042: Status 404 returned error can't find the container with id 2b0620f2b120d71bb09a4b146b409c1a58fceadcd60e7ff0017969199bab2042
Jul 16 21:57:15.165793 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.165771    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:15.244711 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.244687    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"]
Jul 16 21:57:15.247642 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:15.247611    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40f2307_e006_4b84_a5bf_3784f32b595a.slice/crio-b532a7a6480b73ff9a22531dfae1b3a44e7d64210555cd0939afd56aee04031d WatchSource:0}: Error finding container b532a7a6480b73ff9a22531dfae1b3a44e7d64210555cd0939afd56aee04031d: Status 404 returned error can't find the container with id b532a7a6480b73ff9a22531dfae1b3a44e7d64210555cd0939afd56aee04031d
Jul 16 21:57:15.272170 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.272140    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:15.320991 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.320965    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"]
Jul 16 21:57:15.334584 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:15.334547    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd16614_b2c4_4a9c_b1e0_26333d4956e0.slice/crio-82f6d4c041268adf1b8587325252cb67603a49209cc5f4ccd6462d7334d204ef WatchSource:0}: Error finding container 82f6d4c041268adf1b8587325252cb67603a49209cc5f4ccd6462d7334d204ef: Status 404 returned error can't find the container with id 82f6d4c041268adf1b8587325252cb67603a49209cc5f4ccd6462d7334d204ef
Jul 16 21:57:15.432160 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.432124    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"]
Jul 16 21:57:15.460597 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:15.460567    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1721e48a_e57a_494c_b617_a5aa37fff968.slice/crio-be714a14f1b94afa7211afc353bb468e5084fc43fcda7c779d9c2c69cc5c0e39 WatchSource:0}: Error finding container be714a14f1b94afa7211afc353bb468e5084fc43fcda7c779d9c2c69cc5c0e39: Status 404 returned error can't find the container with id be714a14f1b94afa7211afc353bb468e5084fc43fcda7c779d9c2c69cc5c0e39
Jul 16 21:57:15.494060 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.494031    2557 generic.go:356] "Generic (PLEG): container finished" podID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerID="666b633ef6900638883ebe5b668e989f565cc7315f947bb63527bcb783dbb7f9" exitCode=0
Jul 16 21:57:15.494156 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.494111    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn" event={"ID":"644d2f11-9d76-480d-9cc3-86a5de82cd18","Type":"ContainerDied","Data":"666b633ef6900638883ebe5b668e989f565cc7315f947bb63527bcb783dbb7f9"}
Jul 16 21:57:15.494221 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.494205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn" event={"ID":"644d2f11-9d76-480d-9cc3-86a5de82cd18","Type":"ContainerStarted","Data":"2b0620f2b120d71bb09a4b146b409c1a58fceadcd60e7ff0017969199bab2042"}
Jul 16 21:57:15.495890 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.495863    2557 generic.go:356] "Generic (PLEG): container finished" podID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerID="89f5a68e01e63e9829296f9c1c9b56643959ea363b5df47aa56b4ea67ba51582" exitCode=0
Jul 16 21:57:15.495992 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.495937    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w" event={"ID":"c40f2307-e006-4b84-a5bf-3784f32b595a","Type":"ContainerDied","Data":"89f5a68e01e63e9829296f9c1c9b56643959ea363b5df47aa56b4ea67ba51582"}
Jul 16 21:57:15.495992 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.495988    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w" event={"ID":"c40f2307-e006-4b84-a5bf-3784f32b595a","Type":"ContainerStarted","Data":"b532a7a6480b73ff9a22531dfae1b3a44e7d64210555cd0939afd56aee04031d"}
Jul 16 21:57:15.497294 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.497266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf" event={"ID":"1721e48a-e57a-494c-b617-a5aa37fff968","Type":"ContainerStarted","Data":"be714a14f1b94afa7211afc353bb468e5084fc43fcda7c779d9c2c69cc5c0e39"}
Jul 16 21:57:15.499071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.499048    2557 generic.go:356] "Generic (PLEG): container finished" podID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerID="bed76f732da6a83055bb067a1b5b2fe0728e041c8adfec798cec495252138507" exitCode=0
Jul 16 21:57:15.499156 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.499131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l" event={"ID":"fbd16614-b2c4-4a9c-b1e0-26333d4956e0","Type":"ContainerDied","Data":"bed76f732da6a83055bb067a1b5b2fe0728e041c8adfec798cec495252138507"}
Jul 16 21:57:15.499212 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:15.499169    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l" event={"ID":"fbd16614-b2c4-4a9c-b1e0-26333d4956e0","Type":"ContainerStarted","Data":"82f6d4c041268adf1b8587325252cb67603a49209cc5f4ccd6462d7334d204ef"}
Jul 16 21:57:16.506288 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:16.506262    2557 generic.go:356] "Generic (PLEG): container finished" podID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerID="18b33d60e85673e28366a390f1928f017ce4edecda6c5bdd1273e63621854210" exitCode=0
Jul 16 21:57:16.506645 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:16.506337    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w" event={"ID":"c40f2307-e006-4b84-a5bf-3784f32b595a","Type":"ContainerDied","Data":"18b33d60e85673e28366a390f1928f017ce4edecda6c5bdd1273e63621854210"}
Jul 16 21:57:16.507857 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:16.507835    2557 generic.go:356] "Generic (PLEG): container finished" podID="1721e48a-e57a-494c-b617-a5aa37fff968" containerID="3ad91c0852ced47db99974adaefda7625fdad36a1b4669f7960aeb29f3ca5dbc" exitCode=0
Jul 16 21:57:16.507999 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:16.507931    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf" event={"ID":"1721e48a-e57a-494c-b617-a5aa37fff968","Type":"ContainerDied","Data":"3ad91c0852ced47db99974adaefda7625fdad36a1b4669f7960aeb29f3ca5dbc"}
Jul 16 21:57:17.517949 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:17.517919    2557 generic.go:356] "Generic (PLEG): container finished" podID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerID="8cfc4b154f61f5b3c9ce8cae54edd623b7b18081ff3e2f2c9c1adf75a2f85a64" exitCode=0
Jul 16 21:57:17.518410 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:17.517999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn" event={"ID":"644d2f11-9d76-480d-9cc3-86a5de82cd18","Type":"ContainerDied","Data":"8cfc4b154f61f5b3c9ce8cae54edd623b7b18081ff3e2f2c9c1adf75a2f85a64"}
Jul 16 21:57:17.520092 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:17.520069    2557 generic.go:356] "Generic (PLEG): container finished" podID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerID="38a6b9d7e75fb9dc205ce89d062e56b0ed0a60f752c3fc14a552a5abc4ef17c9" exitCode=0
Jul 16 21:57:17.520189 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:17.520144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w" event={"ID":"c40f2307-e006-4b84-a5bf-3784f32b595a","Type":"ContainerDied","Data":"38a6b9d7e75fb9dc205ce89d062e56b0ed0a60f752c3fc14a552a5abc4ef17c9"}
Jul 16 21:57:17.521748 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:17.521729    2557 generic.go:356] "Generic (PLEG): container finished" podID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerID="66d58f86424e1acce5b58a132af9a8c803c80a6fbb339dbb19b7fa004463e693" exitCode=0
Jul 16 21:57:17.521875 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:17.521758    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l" event={"ID":"fbd16614-b2c4-4a9c-b1e0-26333d4956e0","Type":"ContainerDied","Data":"66d58f86424e1acce5b58a132af9a8c803c80a6fbb339dbb19b7fa004463e693"}
Jul 16 21:57:18.528457 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.528419    2557 generic.go:356] "Generic (PLEG): container finished" podID="1721e48a-e57a-494c-b617-a5aa37fff968" containerID="6828ee68d86d3cb97b73626734c7961f5154f63c9209699f6c27e3f763cb9f31" exitCode=0
Jul 16 21:57:18.528923 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.528501    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf" event={"ID":"1721e48a-e57a-494c-b617-a5aa37fff968","Type":"ContainerDied","Data":"6828ee68d86d3cb97b73626734c7961f5154f63c9209699f6c27e3f763cb9f31"}
Jul 16 21:57:18.530785 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.530751    2557 generic.go:356] "Generic (PLEG): container finished" podID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerID="5811b9e8ce6799f2453eb7c4433dc0072eba0bf3aa3b892c231e5c37f57174e5" exitCode=0
Jul 16 21:57:18.530893 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.530837    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l" event={"ID":"fbd16614-b2c4-4a9c-b1e0-26333d4956e0","Type":"ContainerDied","Data":"5811b9e8ce6799f2453eb7c4433dc0072eba0bf3aa3b892c231e5c37f57174e5"}
Jul 16 21:57:18.532935 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.532916    2557 generic.go:356] "Generic (PLEG): container finished" podID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerID="c900e7d1af4bb99f165c4d8b859b3ceccbe90d73a5edd54c0849d733d32332cd" exitCode=0
Jul 16 21:57:18.533033 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.533006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn" event={"ID":"644d2f11-9d76-480d-9cc3-86a5de82cd18","Type":"ContainerDied","Data":"c900e7d1af4bb99f165c4d8b859b3ceccbe90d73a5edd54c0849d733d32332cd"}
Jul 16 21:57:18.679250 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.679223    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:18.714259 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.714233    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-bundle\") pod \"c40f2307-e006-4b84-a5bf-3784f32b595a\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") "
Jul 16 21:57:18.714368 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.714279    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-util\") pod \"c40f2307-e006-4b84-a5bf-3784f32b595a\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") "
Jul 16 21:57:18.714368 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.714300    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh8fz\" (UniqueName: \"kubernetes.io/projected/c40f2307-e006-4b84-a5bf-3784f32b595a-kube-api-access-fh8fz\") pod \"c40f2307-e006-4b84-a5bf-3784f32b595a\" (UID: \"c40f2307-e006-4b84-a5bf-3784f32b595a\") "
Jul 16 21:57:18.714708 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.714681    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-bundle" (OuterVolumeSpecName: "bundle") pod "c40f2307-e006-4b84-a5bf-3784f32b595a" (UID: "c40f2307-e006-4b84-a5bf-3784f32b595a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:18.716885 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.716779    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40f2307-e006-4b84-a5bf-3784f32b595a-kube-api-access-fh8fz" (OuterVolumeSpecName: "kube-api-access-fh8fz") pod "c40f2307-e006-4b84-a5bf-3784f32b595a" (UID: "c40f2307-e006-4b84-a5bf-3784f32b595a"). InnerVolumeSpecName "kube-api-access-fh8fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:18.719796 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.719777    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-util" (OuterVolumeSpecName: "util") pod "c40f2307-e006-4b84-a5bf-3784f32b595a" (UID: "c40f2307-e006-4b84-a5bf-3784f32b595a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:18.815348 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.815265    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:18.815348 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.815293    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40f2307-e006-4b84-a5bf-3784f32b595a-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:18.815348 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:18.815302    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8fz\" (UniqueName: \"kubernetes.io/projected/c40f2307-e006-4b84-a5bf-3784f32b595a-kube-api-access-fh8fz\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:19.539687 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.539661    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w"
Jul 16 21:57:19.539687 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.539663    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efczv7w" event={"ID":"c40f2307-e006-4b84-a5bf-3784f32b595a","Type":"ContainerDied","Data":"b532a7a6480b73ff9a22531dfae1b3a44e7d64210555cd0939afd56aee04031d"}
Jul 16 21:57:19.540172 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.539714    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b532a7a6480b73ff9a22531dfae1b3a44e7d64210555cd0939afd56aee04031d"
Jul 16 21:57:19.541856 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.541833    2557 generic.go:356] "Generic (PLEG): container finished" podID="1721e48a-e57a-494c-b617-a5aa37fff968" containerID="5649ca2599974afb5fcfb254e5ceb9f6c1d91a36e235eb11ee0748ac228b46ae" exitCode=0
Jul 16 21:57:19.541985 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.541912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf" event={"ID":"1721e48a-e57a-494c-b617-a5aa37fff968","Type":"ContainerDied","Data":"5649ca2599974afb5fcfb254e5ceb9f6c1d91a36e235eb11ee0748ac228b46ae"}
Jul 16 21:57:19.687037 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.687012    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:19.716161 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.716133    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:19.724226 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.724198    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm9g4\" (UniqueName: \"kubernetes.io/projected/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-kube-api-access-tm9g4\") pod \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") "
Jul 16 21:57:19.724356 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.724324    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-bundle\") pod \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") "
Jul 16 21:57:19.724417 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.724358    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-util\") pod \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\" (UID: \"fbd16614-b2c4-4a9c-b1e0-26333d4956e0\") "
Jul 16 21:57:19.724929 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.724899    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-bundle" (OuterVolumeSpecName: "bundle") pod "fbd16614-b2c4-4a9c-b1e0-26333d4956e0" (UID: "fbd16614-b2c4-4a9c-b1e0-26333d4956e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:19.726555 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.726523    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-kube-api-access-tm9g4" (OuterVolumeSpecName: "kube-api-access-tm9g4") pod "fbd16614-b2c4-4a9c-b1e0-26333d4956e0" (UID: "fbd16614-b2c4-4a9c-b1e0-26333d4956e0"). InnerVolumeSpecName "kube-api-access-tm9g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:19.729199 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.729164    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-util" (OuterVolumeSpecName: "util") pod "fbd16614-b2c4-4a9c-b1e0-26333d4956e0" (UID: "fbd16614-b2c4-4a9c-b1e0-26333d4956e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:19.825264 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825172    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-util\") pod \"644d2f11-9d76-480d-9cc3-86a5de82cd18\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") "
Jul 16 21:57:19.825264 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825209    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lngw8\" (UniqueName: \"kubernetes.io/projected/644d2f11-9d76-480d-9cc3-86a5de82cd18-kube-api-access-lngw8\") pod \"644d2f11-9d76-480d-9cc3-86a5de82cd18\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") "
Jul 16 21:57:19.825457 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825272    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-bundle\") pod \"644d2f11-9d76-480d-9cc3-86a5de82cd18\" (UID: \"644d2f11-9d76-480d-9cc3-86a5de82cd18\") "
Jul 16 21:57:19.825494 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825476    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm9g4\" (UniqueName: \"kubernetes.io/projected/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-kube-api-access-tm9g4\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:19.825494 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825487    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:19.825552 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825495    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbd16614-b2c4-4a9c-b1e0-26333d4956e0-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:19.825783 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.825752    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-bundle" (OuterVolumeSpecName: "bundle") pod "644d2f11-9d76-480d-9cc3-86a5de82cd18" (UID: "644d2f11-9d76-480d-9cc3-86a5de82cd18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:19.827393 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.827377    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644d2f11-9d76-480d-9cc3-86a5de82cd18-kube-api-access-lngw8" (OuterVolumeSpecName: "kube-api-access-lngw8") pod "644d2f11-9d76-480d-9cc3-86a5de82cd18" (UID: "644d2f11-9d76-480d-9cc3-86a5de82cd18"). InnerVolumeSpecName "kube-api-access-lngw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:19.830672 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.830638    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-util" (OuterVolumeSpecName: "util") pod "644d2f11-9d76-480d-9cc3-86a5de82cd18" (UID: "644d2f11-9d76-480d-9cc3-86a5de82cd18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:19.926367 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.926337    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:19.926367 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.926366    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lngw8\" (UniqueName: \"kubernetes.io/projected/644d2f11-9d76-480d-9cc3-86a5de82cd18-kube-api-access-lngw8\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:19.926522 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:19.926375    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/644d2f11-9d76-480d-9cc3-86a5de82cd18-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:20.549243 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.549218    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn"
Jul 16 21:57:20.549664 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.549252    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36bj5hn" event={"ID":"644d2f11-9d76-480d-9cc3-86a5de82cd18","Type":"ContainerDied","Data":"2b0620f2b120d71bb09a4b146b409c1a58fceadcd60e7ff0017969199bab2042"}
Jul 16 21:57:20.549664 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.549300    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0620f2b120d71bb09a4b146b409c1a58fceadcd60e7ff0017969199bab2042"
Jul 16 21:57:20.551153 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.551126    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l"
Jul 16 21:57:20.551279 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.551182    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab6hq6l" event={"ID":"fbd16614-b2c4-4a9c-b1e0-26333d4956e0","Type":"ContainerDied","Data":"82f6d4c041268adf1b8587325252cb67603a49209cc5f4ccd6462d7334d204ef"}
Jul 16 21:57:20.551279 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.551230    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82f6d4c041268adf1b8587325252cb67603a49209cc5f4ccd6462d7334d204ef"
Jul 16 21:57:20.685940 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.685918    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:20.733772 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.733746    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-bundle\") pod \"1721e48a-e57a-494c-b617-a5aa37fff968\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") "
Jul 16 21:57:20.733926 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.733782    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vzvk\" (UniqueName: \"kubernetes.io/projected/1721e48a-e57a-494c-b617-a5aa37fff968-kube-api-access-4vzvk\") pod \"1721e48a-e57a-494c-b617-a5aa37fff968\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") "
Jul 16 21:57:20.733926 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.733807    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-util\") pod \"1721e48a-e57a-494c-b617-a5aa37fff968\" (UID: \"1721e48a-e57a-494c-b617-a5aa37fff968\") "
Jul 16 21:57:20.734466 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.734439    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-bundle" (OuterVolumeSpecName: "bundle") pod "1721e48a-e57a-494c-b617-a5aa37fff968" (UID: "1721e48a-e57a-494c-b617-a5aa37fff968"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:20.736085 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.736017    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1721e48a-e57a-494c-b617-a5aa37fff968-kube-api-access-4vzvk" (OuterVolumeSpecName: "kube-api-access-4vzvk") pod "1721e48a-e57a-494c-b617-a5aa37fff968" (UID: "1721e48a-e57a-494c-b617-a5aa37fff968"). InnerVolumeSpecName "kube-api-access-4vzvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:20.742495 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.742470    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-util" (OuterVolumeSpecName: "util") pod "1721e48a-e57a-494c-b617-a5aa37fff968" (UID: "1721e48a-e57a-494c-b617-a5aa37fff968"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:57:20.835300 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.835210    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-util\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:20.835300 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.835241    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1721e48a-e57a-494c-b617-a5aa37fff968-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:20.835300 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:20.835250    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vzvk\" (UniqueName: \"kubernetes.io/projected/1721e48a-e57a-494c-b617-a5aa37fff968-kube-api-access-4vzvk\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:57:21.559186 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:21.559160    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf"
Jul 16 21:57:21.563189 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:21.563154    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76f4kbf" event={"ID":"1721e48a-e57a-494c-b617-a5aa37fff968","Type":"ContainerDied","Data":"be714a14f1b94afa7211afc353bb468e5084fc43fcda7c779d9c2c69cc5c0e39"}
Jul 16 21:57:21.563324 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:21.563197    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be714a14f1b94afa7211afc353bb468e5084fc43fcda7c779d9c2c69cc5c0e39"
Jul 16 21:57:26.790177 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790139    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"]
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790493    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="extract"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790504    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="extract"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790518    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="util"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790524    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="util"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790534    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="util"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790539    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="util"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790545    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="util"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790550    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="util"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790556    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="pull"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790561    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="pull"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790566    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="extract"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790571    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="extract"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790579    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="pull"
Jul 16 21:57:26.790579 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790584    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="pull"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790590    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790595    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790602    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="pull"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790607    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="pull"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790614    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="pull"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790620    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="pull"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790626    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="util"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790630    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="util"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790636    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790641    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790689    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd16614-b2c4-4a9c-b1e0-26333d4956e0" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790696    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="644d2f11-9d76-480d-9cc3-86a5de82cd18" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790705    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1721e48a-e57a-494c-b617-a5aa37fff968" containerName="extract"
Jul 16 21:57:26.791075 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.790711    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40f2307-e006-4b84-a5bf-3784f32b595a" containerName="extract"
Jul 16 21:57:26.793879 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.793864    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:26.803734 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.803708    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:57:26.803882 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.803760    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:57:26.818044 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.818009    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"]
Jul 16 21:57:26.822061 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.822039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wvfpd\""
Jul 16 21:57:26.891900 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.891866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f6s\" (UniqueName: \"kubernetes.io/projected/656c1e93-3a69-4ca6-a032-d0c8c1706714-kube-api-access-g6f6s\") pod \"limitador-operator-controller-manager-6f9f468797-ct7rc\" (UID: \"656c1e93-3a69-4ca6-a032-d0c8c1706714\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:26.992948 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:26.992917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f6s\" (UniqueName: \"kubernetes.io/projected/656c1e93-3a69-4ca6-a032-d0c8c1706714-kube-api-access-g6f6s\") pod \"limitador-operator-controller-manager-6f9f468797-ct7rc\" (UID: \"656c1e93-3a69-4ca6-a032-d0c8c1706714\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:27.011112 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:27.011084    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f6s\" (UniqueName: \"kubernetes.io/projected/656c1e93-3a69-4ca6-a032-d0c8c1706714-kube-api-access-g6f6s\") pod \"limitador-operator-controller-manager-6f9f468797-ct7rc\" (UID: \"656c1e93-3a69-4ca6-a032-d0c8c1706714\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:27.103331 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:27.103257    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:27.244485 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:27.244431    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"]
Jul 16 21:57:27.246889 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:27.246858    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656c1e93_3a69_4ca6_a032_d0c8c1706714.slice/crio-32caef3adeae49d4d82eabcb0e453b31c4fb5e7b555d1b0d72a3098f646fbbaa WatchSource:0}: Error finding container 32caef3adeae49d4d82eabcb0e453b31c4fb5e7b555d1b0d72a3098f646fbbaa: Status 404 returned error can't find the container with id 32caef3adeae49d4d82eabcb0e453b31c4fb5e7b555d1b0d72a3098f646fbbaa
Jul 16 21:57:27.592135 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:27.592096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc" event={"ID":"656c1e93-3a69-4ca6-a032-d0c8c1706714","Type":"ContainerStarted","Data":"32caef3adeae49d4d82eabcb0e453b31c4fb5e7b555d1b0d72a3098f646fbbaa"}
Jul 16 21:57:30.611234 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:30.611199    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc" event={"ID":"656c1e93-3a69-4ca6-a032-d0c8c1706714","Type":"ContainerStarted","Data":"32f054b601dd1b695052116dc1dbacdb227213a9da5c45c99057de22c7e8beef"}
Jul 16 21:57:30.611770 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:30.611340    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:30.649226 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:30.649174    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc" podStartSLOduration=2.139947457 podStartE2EDuration="4.649160551s" totalImagesPullingTime="2.509213094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:30.647416375 +0000 UTC m=+461.626765675" watchObservedRunningTime="2026-07-16 21:57:30.649160551 +0000 UTC m=+461.628509851"
Jul 16 21:57:31.877554 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:31.877521    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"]
Jul 16 21:57:31.881188 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:31.881171    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:31.884417 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:31.884398    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5gmpc\""
Jul 16 21:57:31.902920 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:31.899320    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"]
Jul 16 21:57:31.937055 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:31.937028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqrm\" (UniqueName: \"kubernetes.io/projected/8548fc5b-22e5-40d5-9ff9-562ead763bd5-kube-api-access-zlqrm\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-kp78l\" (UID: \"8548fc5b-22e5-40d5-9ff9-562ead763bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:31.937207 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:31.937080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8548fc5b-22e5-40d5-9ff9-562ead763bd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-kp78l\" (UID: \"8548fc5b-22e5-40d5-9ff9-562ead763bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:32.037570 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.037538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqrm\" (UniqueName: \"kubernetes.io/projected/8548fc5b-22e5-40d5-9ff9-562ead763bd5-kube-api-access-zlqrm\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-kp78l\" (UID: \"8548fc5b-22e5-40d5-9ff9-562ead763bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:32.037749 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.037586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8548fc5b-22e5-40d5-9ff9-562ead763bd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-kp78l\" (UID: \"8548fc5b-22e5-40d5-9ff9-562ead763bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:32.037941 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.037925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8548fc5b-22e5-40d5-9ff9-562ead763bd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-kp78l\" (UID: \"8548fc5b-22e5-40d5-9ff9-562ead763bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:32.057735 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.057707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqrm\" (UniqueName: \"kubernetes.io/projected/8548fc5b-22e5-40d5-9ff9-562ead763bd5-kube-api-access-zlqrm\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-kp78l\" (UID: \"8548fc5b-22e5-40d5-9ff9-562ead763bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:32.191214 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.191187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:32.367082 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.367055    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"]
Jul 16 21:57:32.369803 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:32.369778    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8548fc5b_22e5_40d5_9ff9_562ead763bd5.slice/crio-0b0ea696c207872dea5e8a01d9850d5c54027796978b5e3506197287c97f79d2 WatchSource:0}: Error finding container 0b0ea696c207872dea5e8a01d9850d5c54027796978b5e3506197287c97f79d2: Status 404 returned error can't find the container with id 0b0ea696c207872dea5e8a01d9850d5c54027796978b5e3506197287c97f79d2
Jul 16 21:57:32.622461 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:32.622385    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l" event={"ID":"8548fc5b-22e5-40d5-9ff9-562ead763bd5","Type":"ContainerStarted","Data":"0b0ea696c207872dea5e8a01d9850d5c54027796978b5e3506197287c97f79d2"}
Jul 16 21:57:36.359901 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.359867    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"]
Jul 16 21:57:36.368877 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.368847    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:36.371676 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.371654    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fb59n\""
Jul 16 21:57:36.371861 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.371848    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 16 21:57:36.385305 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.385277    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"]
Jul 16 21:57:36.476715 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.476680    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmrc\" (UniqueName: \"kubernetes.io/projected/23f107ac-7611-4768-b8c8-6e92eebead92-kube-api-access-zsmrc\") pod \"dns-operator-controller-manager-65b49595d7-76qtk\" (UID: \"23f107ac-7611-4768-b8c8-6e92eebead92\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:36.577409 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.577368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmrc\" (UniqueName: \"kubernetes.io/projected/23f107ac-7611-4768-b8c8-6e92eebead92-kube-api-access-zsmrc\") pod \"dns-operator-controller-manager-65b49595d7-76qtk\" (UID: \"23f107ac-7611-4768-b8c8-6e92eebead92\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:36.588136 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.588108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmrc\" (UniqueName: \"kubernetes.io/projected/23f107ac-7611-4768-b8c8-6e92eebead92-kube-api-access-zsmrc\") pod \"dns-operator-controller-manager-65b49595d7-76qtk\" (UID: \"23f107ac-7611-4768-b8c8-6e92eebead92\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:36.680526 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.680490    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:36.857976 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:36.857952    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"]
Jul 16 21:57:36.862870 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:36.862839    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f107ac_7611_4768_b8c8_6e92eebead92.slice/crio-b613d97d130dbbcaf89b970c77512c0a106a65b3543adce4a71720473150072c WatchSource:0}: Error finding container b613d97d130dbbcaf89b970c77512c0a106a65b3543adce4a71720473150072c: Status 404 returned error can't find the container with id b613d97d130dbbcaf89b970c77512c0a106a65b3543adce4a71720473150072c
Jul 16 21:57:37.654064 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:37.653964    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk" event={"ID":"23f107ac-7611-4768-b8c8-6e92eebead92","Type":"ContainerStarted","Data":"b613d97d130dbbcaf89b970c77512c0a106a65b3543adce4a71720473150072c"}
Jul 16 21:57:39.668931 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:39.668889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk" event={"ID":"23f107ac-7611-4768-b8c8-6e92eebead92","Type":"ContainerStarted","Data":"3f47d707b2f4c6b801c795c24a992d52f41df3108915084ca46f8c6f3317450e"}
Jul 16 21:57:39.669366 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:39.669021    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:39.693733 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:39.693675    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk" podStartSLOduration=1.068170817 podStartE2EDuration="3.693654198s" totalImagesPullingTime="2.625483381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:39.689988299 +0000 UTC m=+470.669337600" watchObservedRunningTime="2026-07-16 21:57:39.693654198 +0000 UTC m=+470.673003501"
Jul 16 21:57:41.618283 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:41.618242    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-ct7rc"
Jul 16 21:57:41.682534 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:41.682482    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l" event={"ID":"8548fc5b-22e5-40d5-9ff9-562ead763bd5","Type":"ContainerStarted","Data":"5c482c841e18427b2c5141b34cfd24f30cca16d70d4e61217b8dc0e0fae2c3f4"}
Jul 16 21:57:41.682773 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:41.682643    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:57:41.701716 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:41.701658    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l" podStartSLOduration=1.6084564810000002 podStartE2EDuration="10.701637515s" totalImagesPullingTime="9.093181034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:41.701354344 +0000 UTC m=+472.680703643" watchObservedRunningTime="2026-07-16 21:57:41.701637515 +0000 UTC m=+472.680986815"
Jul 16 21:57:43.929308 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:43.929271    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"]
Jul 16 21:57:43.933330 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:43.933297    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:43.935758 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:43.935735    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 21:57:43.935923 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:43.935779    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p86z2\""
Jul 16 21:57:43.935923 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:43.935778    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 21:57:43.942267 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:43.942243    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"]
Jul 16 21:57:44.048697 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.048661    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c484bc34-bb8e-475a-b29b-8afc25416370-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.048889 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.048702    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9m4\" (UniqueName: \"kubernetes.io/projected/c484bc34-bb8e-475a-b29b-8afc25416370-kube-api-access-mz9m4\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.048889 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.048840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c484bc34-bb8e-475a-b29b-8afc25416370-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.149927 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.149890    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9m4\" (UniqueName: \"kubernetes.io/projected/c484bc34-bb8e-475a-b29b-8afc25416370-kube-api-access-mz9m4\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.150111 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.150008    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c484bc34-bb8e-475a-b29b-8afc25416370-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.150111 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.150063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c484bc34-bb8e-475a-b29b-8afc25416370-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.150723 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.150692    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c484bc34-bb8e-475a-b29b-8afc25416370-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.152517 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.152498    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c484bc34-bb8e-475a-b29b-8afc25416370-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.158575 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.158554    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9m4\" (UniqueName: \"kubernetes.io/projected/c484bc34-bb8e-475a-b29b-8afc25416370-kube-api-access-mz9m4\") pod \"kuadrant-console-plugin-85fd4b8b9d-6vqdx\" (UID: \"c484bc34-bb8e-475a-b29b-8afc25416370\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.252928 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.252804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"
Jul 16 21:57:44.420861 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.420678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx"]
Jul 16 21:57:44.423323 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:57:44.423292    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc484bc34_bb8e_475a_b29b_8afc25416370.slice/crio-7bb547ad6110fcbae09b0886d5f1eca5dd166b039fc51a90e3efea1e286bf813 WatchSource:0}: Error finding container 7bb547ad6110fcbae09b0886d5f1eca5dd166b039fc51a90e3efea1e286bf813: Status 404 returned error can't find the container with id 7bb547ad6110fcbae09b0886d5f1eca5dd166b039fc51a90e3efea1e286bf813
Jul 16 21:57:44.706904 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:44.706865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx" event={"ID":"c484bc34-bb8e-475a-b29b-8afc25416370","Type":"ContainerStarted","Data":"7bb547ad6110fcbae09b0886d5f1eca5dd166b039fc51a90e3efea1e286bf813"}
Jul 16 21:57:49.736807 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:49.736770    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx" event={"ID":"c484bc34-bb8e-475a-b29b-8afc25416370","Type":"ContainerStarted","Data":"8ea4886b1c1a36196d3fa6511a8140129845f378bc29b9bb76d53b8a912cad4b"}
Jul 16 21:57:49.759433 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:49.759387    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-6vqdx" podStartSLOduration=2.090270044 podStartE2EDuration="6.759372465s" totalImagesPullingTime="4.669102421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:49.758033573 +0000 UTC m=+480.737382871" watchObservedRunningTime="2026-07-16 21:57:49.759372465 +0000 UTC m=+480.738721763"
Jul 16 21:57:50.677496 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:50.677459    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-76qtk"
Jul 16 21:57:52.696244 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:57:52.696217    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-kp78l"
Jul 16 21:58:24.438866 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.438832    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-4xqs4"]
Jul 16 21:58:24.458552 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.458524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-4xqs4"]
Jul 16 21:58:24.458696 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.458630    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.462016 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.461990    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 21:58:24.473873 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.473850    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-4xqs4"]
Jul 16 21:58:24.558194 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.558174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da4478a9-b81e-4d50-a483-b876b7b90147-config-file\") pod \"limitador-limitador-69574b596d-4xqs4\" (UID: \"da4478a9-b81e-4d50-a483-b876b7b90147\") " pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.558315 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.558212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77sfp\" (UniqueName: \"kubernetes.io/projected/da4478a9-b81e-4d50-a483-b876b7b90147-kube-api-access-77sfp\") pod \"limitador-limitador-69574b596d-4xqs4\" (UID: \"da4478a9-b81e-4d50-a483-b876b7b90147\") " pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.658841 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.658798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da4478a9-b81e-4d50-a483-b876b7b90147-config-file\") pod \"limitador-limitador-69574b596d-4xqs4\" (UID: \"da4478a9-b81e-4d50-a483-b876b7b90147\") " pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.658952 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.658861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77sfp\" (UniqueName: \"kubernetes.io/projected/da4478a9-b81e-4d50-a483-b876b7b90147-kube-api-access-77sfp\") pod \"limitador-limitador-69574b596d-4xqs4\" (UID: \"da4478a9-b81e-4d50-a483-b876b7b90147\") " pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.659377 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.659360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da4478a9-b81e-4d50-a483-b876b7b90147-config-file\") pod \"limitador-limitador-69574b596d-4xqs4\" (UID: \"da4478a9-b81e-4d50-a483-b876b7b90147\") " pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.669568 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.669549    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77sfp\" (UniqueName: \"kubernetes.io/projected/da4478a9-b81e-4d50-a483-b876b7b90147-kube-api-access-77sfp\") pod \"limitador-limitador-69574b596d-4xqs4\" (UID: \"da4478a9-b81e-4d50-a483-b876b7b90147\") " pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.768791 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.768732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:24.921607 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:24.921584    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-4xqs4"]
Jul 16 21:58:24.923338 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:58:24.923301    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4478a9_b81e_4d50_a483_b876b7b90147.slice/crio-da91ba8a6313403ff3daa0fb9f303aed846f7a0b8631a9ec2e073c120591c81e WatchSource:0}: Error finding container da91ba8a6313403ff3daa0fb9f303aed846f7a0b8631a9ec2e073c120591c81e: Status 404 returned error can't find the container with id da91ba8a6313403ff3daa0fb9f303aed846f7a0b8631a9ec2e073c120591c81e
Jul 16 21:58:25.924364 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:25.924334    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4" event={"ID":"da4478a9-b81e-4d50-a483-b876b7b90147","Type":"ContainerStarted","Data":"da91ba8a6313403ff3daa0fb9f303aed846f7a0b8631a9ec2e073c120591c81e"}
Jul 16 21:58:28.944248 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:28.944221    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4" event={"ID":"da4478a9-b81e-4d50-a483-b876b7b90147","Type":"ContainerStarted","Data":"8f334057443eddca463e34f8b8d4261f54c84457b98f43f5a25d553e1c32c33d"}
Jul 16 21:58:28.944643 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:28.944375    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:28.962578 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:28.962536    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4" podStartSLOduration=1.331480309 podStartE2EDuration="4.962524194s" totalImagesPullingTime="3.631043885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:28.960836152 +0000 UTC m=+519.940185454" watchObservedRunningTime="2026-07-16 21:58:28.962524194 +0000 UTC m=+519.941873493"
Jul 16 21:58:39.951985 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:39.951955    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-4xqs4"
Jul 16 21:58:58.053726 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.053685    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-5hk5f"]
Jul 16 21:58:58.054287 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.053914    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f" podUID="64809f09-8938-4fcb-88ac-64f0ae3cf640" containerName="manager" containerID="cri-o://d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541" gracePeriod=10
Jul 16 21:58:58.082721 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.082691    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-rptsm"]
Jul 16 21:58:58.086051 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.086030    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.096172 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.096142    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-rptsm"]
Jul 16 21:58:58.234441 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.234400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c-cert\") pod \"kserve-controller-manager-84cf6d888d-rptsm\" (UID: \"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c\") " pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.234604 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.234497    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqddn\" (UniqueName: \"kubernetes.io/projected/1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c-kube-api-access-lqddn\") pod \"kserve-controller-manager-84cf6d888d-rptsm\" (UID: \"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c\") " pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.307285 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.307213    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:58:58.335407 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.335372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c-cert\") pod \"kserve-controller-manager-84cf6d888d-rptsm\" (UID: \"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c\") " pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.335608 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.335430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqddn\" (UniqueName: \"kubernetes.io/projected/1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c-kube-api-access-lqddn\") pod \"kserve-controller-manager-84cf6d888d-rptsm\" (UID: \"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c\") " pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.338344 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.338310    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c-cert\") pod \"kserve-controller-manager-84cf6d888d-rptsm\" (UID: \"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c\") " pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.345197 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.345154    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqddn\" (UniqueName: \"kubernetes.io/projected/1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c-kube-api-access-lqddn\") pod \"kserve-controller-manager-84cf6d888d-rptsm\" (UID: \"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c\") " pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.435869 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.435805    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phfhm\" (UniqueName: \"kubernetes.io/projected/64809f09-8938-4fcb-88ac-64f0ae3cf640-kube-api-access-phfhm\") pod \"64809f09-8938-4fcb-88ac-64f0ae3cf640\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") "
Jul 16 21:58:58.436037 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.435907    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64809f09-8938-4fcb-88ac-64f0ae3cf640-cert\") pod \"64809f09-8938-4fcb-88ac-64f0ae3cf640\" (UID: \"64809f09-8938-4fcb-88ac-64f0ae3cf640\") "
Jul 16 21:58:58.438209 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.438181    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64809f09-8938-4fcb-88ac-64f0ae3cf640-cert" (OuterVolumeSpecName: "cert") pod "64809f09-8938-4fcb-88ac-64f0ae3cf640" (UID: "64809f09-8938-4fcb-88ac-64f0ae3cf640"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:58:58.438321 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.438228    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64809f09-8938-4fcb-88ac-64f0ae3cf640-kube-api-access-phfhm" (OuterVolumeSpecName: "kube-api-access-phfhm") pod "64809f09-8938-4fcb-88ac-64f0ae3cf640" (UID: "64809f09-8938-4fcb-88ac-64f0ae3cf640"). InnerVolumeSpecName "kube-api-access-phfhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:58:58.443383 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.443363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:58.537445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.537407    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phfhm\" (UniqueName: \"kubernetes.io/projected/64809f09-8938-4fcb-88ac-64f0ae3cf640-kube-api-access-phfhm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:58:58.537445 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.537440    2557 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64809f09-8938-4fcb-88ac-64f0ae3cf640-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 21:58:58.578182 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:58.578156    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-rptsm"]
Jul 16 21:58:58.579615 ip-10-0-136-80 kubenswrapper[2557]: W0716 21:58:58.579594    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afc23cb_0744_4cfd_a0fd_3eeb45dfc56c.slice/crio-fb9854e5c80cdedfc507ea85dabee2146181ca4c8dc9549bc9de02c717e5adbd WatchSource:0}: Error finding container fb9854e5c80cdedfc507ea85dabee2146181ca4c8dc9549bc9de02c717e5adbd: Status 404 returned error can't find the container with id fb9854e5c80cdedfc507ea85dabee2146181ca4c8dc9549bc9de02c717e5adbd
Jul 16 21:58:59.100071 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.100039    2557 generic.go:356] "Generic (PLEG): container finished" podID="64809f09-8938-4fcb-88ac-64f0ae3cf640" containerID="d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541" exitCode=0
Jul 16 21:58:59.100539 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.100087    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f"
Jul 16 21:58:59.100539 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.100125    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f" event={"ID":"64809f09-8938-4fcb-88ac-64f0ae3cf640","Type":"ContainerDied","Data":"d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541"}
Jul 16 21:58:59.100539 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.100185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-5hk5f" event={"ID":"64809f09-8938-4fcb-88ac-64f0ae3cf640","Type":"ContainerDied","Data":"785e78dc93799cda0e2c81920fa30a7d58fbabf77ca7061d209e2044c9dd9bc6"}
Jul 16 21:58:59.100539 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.100220    2557 scope.go:117] "RemoveContainer" containerID="d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541"
Jul 16 21:58:59.101854 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.101804    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm" event={"ID":"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c","Type":"ContainerStarted","Data":"542ed32ba79aa7c1b510dfdd27c0038c344bfa06474bcbbd1687aab809c684df"}
Jul 16 21:58:59.101948 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.101881    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm" event={"ID":"1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c","Type":"ContainerStarted","Data":"fb9854e5c80cdedfc507ea85dabee2146181ca4c8dc9549bc9de02c717e5adbd"}
Jul 16 21:58:59.102038 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.102023    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:58:59.110105 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.110085    2557 scope.go:117] "RemoveContainer" containerID="d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541"
Jul 16 21:58:59.110397 ip-10-0-136-80 kubenswrapper[2557]: E0716 21:58:59.110379    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541\": container with ID starting with d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541 not found: ID does not exist" containerID="d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541"
Jul 16 21:58:59.110454 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.110408    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541"} err="failed to get container status \"d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541\": rpc error: code = NotFound desc = could not find container \"d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541\": container with ID starting with d84004e978a2976311661cfdde975c9acbff4e7d3e04e91ab3a590f7500e1541 not found: ID does not exist"
Jul 16 21:58:59.122226 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.122175    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm" podStartSLOduration=0.800283251 podStartE2EDuration="1.122158657s" totalImagesPullingTime="321.875406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:59.119786085 +0000 UTC m=+550.099135375" watchObservedRunningTime="2026-07-16 21:58:59.122158657 +0000 UTC m=+550.101507955"
Jul 16 21:58:59.134488 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.134450    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-5hk5f"]
Jul 16 21:58:59.136661 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.136637    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-5hk5f"]
Jul 16 21:58:59.563655 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:58:59.563617    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64809f09-8938-4fcb-88ac-64f0ae3cf640" path="/var/lib/kubelet/pods/64809f09-8938-4fcb-88ac-64f0ae3cf640/volumes"
Jul 16 21:59:30.112802 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:59:30.112765    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84cf6d888d-rptsm"
Jul 16 21:59:49.468424 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:59:49.468394    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 21:59:49.472332 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:59:49.472294    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 21:59:49.473188 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:59:49.473156    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 21:59:49.477217 ip-10-0-136-80 kubenswrapper[2557]: I0716 21:59:49.477194    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:01:38.763213 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.763180    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"]
Jul 16 22:01:38.763617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.763582    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64809f09-8938-4fcb-88ac-64f0ae3cf640" containerName="manager"
Jul 16 22:01:38.763617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.763597    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="64809f09-8938-4fcb-88ac-64f0ae3cf640" containerName="manager"
Jul 16 22:01:38.763724 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.763668    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="64809f09-8938-4fcb-88ac-64f0ae3cf640" containerName="manager"
Jul 16 22:01:38.766093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.766078    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.779312 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.779289    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:01:38.780341 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.780323    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:01:38.780427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.780351    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 16 22:01:38.780427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.780420    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:01:38.780535 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.780471    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:01:38.793667 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.793639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"]
Jul 16 22:01:38.847490 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.847607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847512    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tls-certs\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.847607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.847607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-model-cache\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.847742 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847687    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-home\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.847742 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847711    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vnz\" (UniqueName: \"kubernetes.io/projected/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kube-api-access-24vnz\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.847874 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.847770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-dshm\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-home\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948337 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948230    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vnz\" (UniqueName: \"kubernetes.io/projected/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kube-api-access-24vnz\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948337 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-dshm\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948337 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948270    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948337 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948313    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tls-certs\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948530 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948530 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-model-cache\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948635 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948611    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-home\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948754 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948898 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948778    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.948898 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.948805    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-model-cache\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.950632 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.950613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-dshm\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.950934 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.950918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tls-certs\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:38.961444 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:38.961425    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vnz\" (UniqueName: \"kubernetes.io/projected/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kube-api-access-24vnz\") pod \"precise-prefix-cache-test-kserve-7597885744-94nl8\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:39.075836 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:39.075766    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:39.211605 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:39.211581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"]
Jul 16 22:01:39.212725 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:01:39.212698    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03a0eac_df9a_4db7_9492_31ab3c2ec013.slice/crio-290c1bd603b35148858bef7d679de6ff2b827c54629ee57aead496c77674cb7c WatchSource:0}: Error finding container 290c1bd603b35148858bef7d679de6ff2b827c54629ee57aead496c77674cb7c: Status 404 returned error can't find the container with id 290c1bd603b35148858bef7d679de6ff2b827c54629ee57aead496c77674cb7c
Jul 16 22:01:39.214533 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:39.214518    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:01:39.948789 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:39.948746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" event={"ID":"d03a0eac-df9a-4db7-9492-31ab3c2ec013","Type":"ContainerStarted","Data":"290c1bd603b35148858bef7d679de6ff2b827c54629ee57aead496c77674cb7c"}
Jul 16 22:01:42.969367 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:42.969329    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" event={"ID":"d03a0eac-df9a-4db7-9492-31ab3c2ec013","Type":"ContainerStarted","Data":"cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65"}
Jul 16 22:01:45.985177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:45.985141    2557 generic.go:356] "Generic (PLEG): container finished" podID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerID="cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65" exitCode=0
Jul 16 22:01:45.985495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:45.985208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" event={"ID":"d03a0eac-df9a-4db7-9492-31ab3c2ec013","Type":"ContainerDied","Data":"cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65"}
Jul 16 22:01:47.996852 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:47.996801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" event={"ID":"d03a0eac-df9a-4db7-9492-31ab3c2ec013","Type":"ContainerStarted","Data":"df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f"}
Jul 16 22:01:48.024060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:48.024015    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" podStartSLOduration=2.688110828 podStartE2EDuration="10.024002665s" totalImagesPullingTime="4.304133989s" totalInitContainerRuntime="3.031757848s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:48.020970992 +0000 UTC m=+719.000320292" watchObservedRunningTime="2026-07-16 22:01:48.024002665 +0000 UTC m=+719.003351962"
Jul 16 22:01:48.076222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:48.076199    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:49.075997 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:49.075968    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:49.088372 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:49.088346    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:01:50.017757 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:01:50.017690    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:02:28.589353 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.589322    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"]
Jul 16 22:02:28.592177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.592155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.595018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.594995    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 16 22:02:28.604587 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.604562    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"]
Jul 16 22:02:28.642409 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642376    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-tmp-dir\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.642576 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmgd\" (UniqueName: \"kubernetes.io/projected/7637673d-8185-4f21-859c-152a90a5b1aa-kube-api-access-klmgd\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.642576 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-model-cache\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.642576 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7637673d-8185-4f21-859c-152a90a5b1aa-tls-certs\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.642695 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642590    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-kserve-provision-location\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.642695 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642626    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-home\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.642695 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.642653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-dshm\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743395 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-dshm\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-tmp-dir\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmgd\" (UniqueName: \"kubernetes.io/projected/7637673d-8185-4f21-859c-152a90a5b1aa-kube-api-access-klmgd\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-model-cache\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743513    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7637673d-8185-4f21-859c-152a90a5b1aa-tls-certs\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-kserve-provision-location\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.743617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.743572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-home\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.744500 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.744327    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-tmp-dir\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.744500 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.744415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-model-cache\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.746408 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.744859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-kserve-provision-location\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.746408 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.745019    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-home\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.746596 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.746575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-dshm\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.754112 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.754092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7637673d-8185-4f21-859c-152a90a5b1aa-tls-certs\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.754265 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.754243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmgd\" (UniqueName: \"kubernetes.io/projected/7637673d-8185-4f21-859c-152a90a5b1aa-kube-api-access-klmgd\") pod \"auth-enabled-test-kserve-598dd9464-dqmnc\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:28.903544 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:28.903441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:02:29.044706 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:29.044658    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"]
Jul 16 22:02:29.046680 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:02:29.046652    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7637673d_8185_4f21_859c_152a90a5b1aa.slice/crio-f55998c3ee7960825254b93bbe1287bcabe9ac5dca2d68e1271a7843a5ecc181 WatchSource:0}: Error finding container f55998c3ee7960825254b93bbe1287bcabe9ac5dca2d68e1271a7843a5ecc181: Status 404 returned error can't find the container with id f55998c3ee7960825254b93bbe1287bcabe9ac5dca2d68e1271a7843a5ecc181
Jul 16 22:02:29.217250 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:29.217164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" event={"ID":"7637673d-8185-4f21-859c-152a90a5b1aa","Type":"ContainerStarted","Data":"503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f"}
Jul 16 22:02:29.217250 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:29.217233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" event={"ID":"7637673d-8185-4f21-859c-152a90a5b1aa","Type":"ContainerStarted","Data":"f55998c3ee7960825254b93bbe1287bcabe9ac5dca2d68e1271a7843a5ecc181"}
Jul 16 22:02:32.234964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:32.234930    2557 generic.go:356] "Generic (PLEG): container finished" podID="7637673d-8185-4f21-859c-152a90a5b1aa" containerID="503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f" exitCode=0
Jul 16 22:02:32.235384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:32.234970    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" event={"ID":"7637673d-8185-4f21-859c-152a90a5b1aa","Type":"ContainerDied","Data":"503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f"}
Jul 16 22:02:59.432213 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:59.432175    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" event={"ID":"7637673d-8185-4f21-859c-152a90a5b1aa","Type":"ContainerStarted","Data":"500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52"}
Jul 16 22:02:59.456134 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:59.456082    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podStartSLOduration=1.489823547 podStartE2EDuration="31.45606752s" totalImagesPullingTime="27.017764921s" totalInitContainerRuntime="2.948479052s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:59.453043182 +0000 UTC m=+790.432392482" watchObservedRunningTime="2026-07-16 22:02:59.45606752 +0000 UTC m=+790.435416818"
Jul 16 22:02:59.904146 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:02:59.904110    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:03:08.903997 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:08.903960    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:03:08.905606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:08.905580    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:03:18.903933 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:18.903833    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:03:27.768103 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:27.768061    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"]
Jul 16 22:03:28.904859 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:28.904786    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:03:38.903936 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:38.903889    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:03:42.788709 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:42.788652    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="main" containerID="cri-o://df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f" gracePeriod=45
Jul 16 22:03:43.038461 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.038437    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:03:43.143162 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143135    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kserve-provision-location\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tmp-dir\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143224    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tls-certs\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143245    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-model-cache\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-dshm\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143322    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24vnz\" (UniqueName: \"kubernetes.io/projected/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kube-api-access-24vnz\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143393    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-home\") pod \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\" (UID: \"d03a0eac-df9a-4db7-9492-31ab3c2ec013\") "
Jul 16 22:03:43.143599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143485    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-model-cache" (OuterVolumeSpecName: "model-cache") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.143599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143514    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.143747 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143711    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.143747 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143729    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.143961 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.143933    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-home" (OuterVolumeSpecName: "home") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.145904 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.145874    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-dshm" (OuterVolumeSpecName: "dshm") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.145904 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.145878    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:03:43.146040 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.146009    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kube-api-access-24vnz" (OuterVolumeSpecName: "kube-api-access-24vnz") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "kube-api-access-24vnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:03:43.196845 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.196794    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d03a0eac-df9a-4db7-9492-31ab3c2ec013" (UID: "d03a0eac-df9a-4db7-9492-31ab3c2ec013"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.245011 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.244977    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.245011 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.245006    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.245011 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.245017    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03a0eac-df9a-4db7-9492-31ab3c2ec013-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.245221 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.245025    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03a0eac-df9a-4db7-9492-31ab3c2ec013-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.245221 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.245035    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vnz\" (UniqueName: \"kubernetes.io/projected/d03a0eac-df9a-4db7-9492-31ab3c2ec013-kube-api-access-24vnz\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.682716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.682680    2557 generic.go:356] "Generic (PLEG): container finished" podID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerID="df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f" exitCode=0
Jul 16 22:03:43.682912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.682729    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"
Jul 16 22:03:43.682912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.682751    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" event={"ID":"d03a0eac-df9a-4db7-9492-31ab3c2ec013","Type":"ContainerDied","Data":"df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f"}
Jul 16 22:03:43.682912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.682826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" event={"ID":"d03a0eac-df9a-4db7-9492-31ab3c2ec013","Type":"ContainerDied","Data":"290c1bd603b35148858bef7d679de6ff2b827c54629ee57aead496c77674cb7c"}
Jul 16 22:03:43.682912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.682856    2557 scope.go:117] "RemoveContainer" containerID="df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f"
Jul 16 22:03:43.693007 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.692987    2557 scope.go:117] "RemoveContainer" containerID="cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65"
Jul 16 22:03:43.701535 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.701515    2557 scope.go:117] "RemoveContainer" containerID="df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f"
Jul 16 22:03:43.701854 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:03:43.701828    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f\": container with ID starting with df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f not found: ID does not exist" containerID="df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f"
Jul 16 22:03:43.701933 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.701866    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f"} err="failed to get container status \"df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f\": rpc error: code = NotFound desc = could not find container \"df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f\": container with ID starting with df9147bf69acdefde16791c0d144982e407a7d48a17db6f284fe3672c46be90f not found: ID does not exist"
Jul 16 22:03:43.701933 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.701885    2557 scope.go:117] "RemoveContainer" containerID="cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65"
Jul 16 22:03:43.702209 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:03:43.702187    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65\": container with ID starting with cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65 not found: ID does not exist" containerID="cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65"
Jul 16 22:03:43.702299 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.702216    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65"} err="failed to get container status \"cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65\": rpc error: code = NotFound desc = could not find container \"cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65\": container with ID starting with cd9ea11655c26017f13eb89a7d59e8c55acdf71f91c72732bbe7ce3b1a1fbc65 not found: ID does not exist"
Jul 16 22:03:43.703890 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.703868    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"]
Jul 16 22:03:43.708373 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:43.708352    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8"]
Jul 16 22:03:44.008289 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:44.008243    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-94nl8" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:03:45.562854 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:45.562794    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" path="/var/lib/kubelet/pods/d03a0eac-df9a-4db7-9492-31ab3c2ec013/volumes"
Jul 16 22:03:48.904059 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:48.904019    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:03:58.904299 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:03:58.904251    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:04:08.904466 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:08.904424    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:04:18.904136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:18.904095    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:04:28.904680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:28.904639    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:04:38.904861 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:38.904804    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:04:48.914297 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:48.914269    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:04:48.921779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:48.921761    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:04:49.506999 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:49.506969    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:04:49.508240 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:49.508218    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:04:49.511309 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:49.511290    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:04:49.512616 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:49.512597    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:04:53.824994 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:04:53.824957    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"]
Jul 16 22:05:08.856074 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:08.856014    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" containerID="cri-o://500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52" gracePeriod=45
Jul 16 22:05:08.915206 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:08.915169    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 16 22:05:09.541969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.541944    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:05:09.597176 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597148    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-kserve-provision-location\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597319 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597185    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-model-cache\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597319 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597242    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7637673d-8185-4f21-859c-152a90a5b1aa-tls-certs\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597319 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597269    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klmgd\" (UniqueName: \"kubernetes.io/projected/7637673d-8185-4f21-859c-152a90a5b1aa-kube-api-access-klmgd\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597319 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597297    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-home\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597542 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597323    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-tmp-dir\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597542 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597344    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-dshm\") pod \"7637673d-8185-4f21-859c-152a90a5b1aa\" (UID: \"7637673d-8185-4f21-859c-152a90a5b1aa\") "
Jul 16 22:05:09.597660 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.597601    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.598785 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.598103    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-home" (OuterVolumeSpecName: "home") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.600249 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.600214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7637673d-8185-4f21-859c-152a90a5b1aa-kube-api-access-klmgd" (OuterVolumeSpecName: "kube-api-access-klmgd") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "kube-api-access-klmgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:05:09.600370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.600304    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7637673d-8185-4f21-859c-152a90a5b1aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:05:09.601643 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.601619    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-dshm" (OuterVolumeSpecName: "dshm") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.611166 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.611116    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.669442 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.669405    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7637673d-8185-4f21-859c-152a90a5b1aa" (UID: "7637673d-8185-4f21-859c-152a90a5b1aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.698119 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698092    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7637673d-8185-4f21-859c-152a90a5b1aa-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.698119 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698120    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmgd\" (UniqueName: \"kubernetes.io/projected/7637673d-8185-4f21-859c-152a90a5b1aa-kube-api-access-klmgd\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.698272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698132    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.698272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698143    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.698272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698153    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.698272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698163    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.698272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:09.698174    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7637673d-8185-4f21-859c-152a90a5b1aa-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:05:10.157053 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.157018    2557 generic.go:356] "Generic (PLEG): container finished" podID="7637673d-8185-4f21-859c-152a90a5b1aa" containerID="500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52" exitCode=0
Jul 16 22:05:10.157443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.157095    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"
Jul 16 22:05:10.157443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.157101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" event={"ID":"7637673d-8185-4f21-859c-152a90a5b1aa","Type":"ContainerDied","Data":"500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52"}
Jul 16 22:05:10.157443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.157155    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc" event={"ID":"7637673d-8185-4f21-859c-152a90a5b1aa","Type":"ContainerDied","Data":"f55998c3ee7960825254b93bbe1287bcabe9ac5dca2d68e1271a7843a5ecc181"}
Jul 16 22:05:10.157443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.157177    2557 scope.go:117] "RemoveContainer" containerID="500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52"
Jul 16 22:05:10.167547 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.167527    2557 scope.go:117] "RemoveContainer" containerID="503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f"
Jul 16 22:05:10.175761 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.175746    2557 scope.go:117] "RemoveContainer" containerID="500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52"
Jul 16 22:05:10.176048 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:05:10.176032    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52\": container with ID starting with 500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52 not found: ID does not exist" containerID="500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52"
Jul 16 22:05:10.176105 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.176078    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52"} err="failed to get container status \"500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52\": rpc error: code = NotFound desc = could not find container \"500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52\": container with ID starting with 500d5cb7738f6da06544c177edf1f9fb3f772e46a39f68158bbbff0337970b52 not found: ID does not exist"
Jul 16 22:05:10.176105 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.176095    2557 scope.go:117] "RemoveContainer" containerID="503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f"
Jul 16 22:05:10.176377 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:05:10.176353    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f\": container with ID starting with 503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f not found: ID does not exist" containerID="503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f"
Jul 16 22:05:10.176463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.176377    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f"} err="failed to get container status \"503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f\": rpc error: code = NotFound desc = could not find container \"503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f\": container with ID starting with 503d0c08173e4812b00eb023e39fedd0590404c9f43c4d916b5257b46a571d8f not found: ID does not exist"
Jul 16 22:05:10.185164 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.185143    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"]
Jul 16 22:05:10.187204 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:10.187185    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-598dd9464-dqmnc"]
Jul 16 22:05:11.563352 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:11.563316    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" path="/var/lib/kubelet/pods/7637673d-8185-4f21-859c-152a90a5b1aa/volumes"
Jul 16 22:05:16.696098 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696065    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"]
Jul 16 22:05:16.696594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696576    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="storage-initializer"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696597    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="storage-initializer"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696609    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="main"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="main"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696637    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="storage-initializer"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696648    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="storage-initializer"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696665    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main"
Jul 16 22:05:16.696681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696673    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main"
Jul 16 22:05:16.697027 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696778    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03a0eac-df9a-4db7-9492-31ab3c2ec013" containerName="main"
Jul 16 22:05:16.697027 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.696795    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7637673d-8185-4f21-859c-152a90a5b1aa" containerName="main"
Jul 16 22:05:16.700064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.700043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.703108 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.703088    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:05:16.703108 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.703091    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:05:16.703273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.703122    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:05:16.703273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.703095    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 16 22:05:16.703559 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.703545    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:05:16.715452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.715426    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"]
Jul 16 22:05:16.859743 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.859711    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-model-cache\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.859743 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.859745    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-home\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.859994 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.859763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlbc\" (UniqueName: \"kubernetes.io/projected/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kube-api-access-pvlbc\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.859994 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.859909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.859994 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.859962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-dshm\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.860145 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.860000    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tmp-dir\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.860145 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.860063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tls-certs\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.948509 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.948436    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"]
Jul 16 22:05:16.951931 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.951914    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:16.954329 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.954312    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-rvgtq\""
Jul 16 22:05:16.960454 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tmp-dir\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960540 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tls-certs\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-model-cache\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-home\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960675 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960597    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlbc\" (UniqueName: \"kubernetes.io/projected/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kube-api-access-pvlbc\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960675 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960765 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960704    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-dshm\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960851 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960836    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tmp-dir\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.960943 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960919    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-model-cache\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.961013 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.960943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-home\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.961069 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.961007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.963274 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.963249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-dshm\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.963627 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.963607    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tls-certs\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:16.965181 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.965160    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"]
Jul 16 22:05:16.972135 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:16.972113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlbc\" (UniqueName: \"kubernetes.io/projected/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kube-api-access-pvlbc\") pod \"auth-invalid-token-test-kserve-6c78d67c56-sxd27\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:17.010680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.010653    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:17.061480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.061294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.061480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.061340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bcf8e4-9551-4999-a508-ac8d7443af71-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.061480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.061370    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pccqf\" (UniqueName: \"kubernetes.io/projected/57bcf8e4-9551-4999-a508-ac8d7443af71-kube-api-access-pccqf\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.061480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.061423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.159853 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.159828    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"]
Jul 16 22:05:17.161275 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:05:17.161248    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3ef3c9_7687_4da2_a037_a833a67c9c0d.slice/crio-5023ec090168d25fe4f6e0ad26ba61e58b0f1be980ec639971d8381a7914c0f5 WatchSource:0}: Error finding container 5023ec090168d25fe4f6e0ad26ba61e58b0f1be980ec639971d8381a7914c0f5: Status 404 returned error can't find the container with id 5023ec090168d25fe4f6e0ad26ba61e58b0f1be980ec639971d8381a7914c0f5
Jul 16 22:05:17.162284 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.162264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.162393 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.162299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.162393 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.162328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bcf8e4-9551-4999-a508-ac8d7443af71-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.162393 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.162368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pccqf\" (UniqueName: \"kubernetes.io/projected/57bcf8e4-9551-4999-a508-ac8d7443af71-kube-api-access-pccqf\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.162695 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.162677    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.162783 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.162673    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.165019 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.165000    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bcf8e4-9551-4999-a508-ac8d7443af71-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.174517 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.174497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pccqf\" (UniqueName: \"kubernetes.io/projected/57bcf8e4-9551-4999-a508-ac8d7443af71-kube-api-access-pccqf\") pod \"auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.195123 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.195103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" event={"ID":"5f3ef3c9-7687-4da2-a037-a833a67c9c0d","Type":"ContainerStarted","Data":"5023ec090168d25fe4f6e0ad26ba61e58b0f1be980ec639971d8381a7914c0f5"}
Jul 16 22:05:17.260944 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.260915    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:17.401964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:17.401940    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"]
Jul 16 22:05:17.403178 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:05:17.403152    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bcf8e4_9551_4999_a508_ac8d7443af71.slice/crio-2941ea7762ec3316e00c39e5726b6811b320de0a81e38ac1715f359abcf5a91f WatchSource:0}: Error finding container 2941ea7762ec3316e00c39e5726b6811b320de0a81e38ac1715f359abcf5a91f: Status 404 returned error can't find the container with id 2941ea7762ec3316e00c39e5726b6811b320de0a81e38ac1715f359abcf5a91f
Jul 16 22:05:18.203481 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:18.203450    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" event={"ID":"57bcf8e4-9551-4999-a508-ac8d7443af71","Type":"ContainerStarted","Data":"a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe"}
Jul 16 22:05:18.203875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:18.203535    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" event={"ID":"57bcf8e4-9551-4999-a508-ac8d7443af71","Type":"ContainerStarted","Data":"2941ea7762ec3316e00c39e5726b6811b320de0a81e38ac1715f359abcf5a91f"}
Jul 16 22:05:18.205383 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:18.205360    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" event={"ID":"5f3ef3c9-7687-4da2-a037-a833a67c9c0d","Type":"ContainerStarted","Data":"1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33"}
Jul 16 22:05:19.214597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:19.214551    2557 generic.go:356] "Generic (PLEG): container finished" podID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerID="a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe" exitCode=0
Jul 16 22:05:19.215094 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:19.214651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" event={"ID":"57bcf8e4-9551-4999-a508-ac8d7443af71","Type":"ContainerDied","Data":"a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe"}
Jul 16 22:05:22.234101 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:22.234068    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" event={"ID":"57bcf8e4-9551-4999-a508-ac8d7443af71","Type":"ContainerStarted","Data":"6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224"}
Jul 16 22:05:22.234557 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:22.234164    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:22.235738 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:22.235710    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerID="1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33" exitCode=0
Jul 16 22:05:22.235738 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:22.235738    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" event={"ID":"5f3ef3c9-7687-4da2-a037-a833a67c9c0d","Type":"ContainerDied","Data":"1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33"}
Jul 16 22:05:22.272535 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:22.272489    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" podStartSLOduration=2.463715485 podStartE2EDuration="6.272475128s" totalImagesPullingTime="3.086216182s" totalInitContainerRuntime="722.543461ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:05:22.269094651 +0000 UTC m=+933.248443949" watchObservedRunningTime="2026-07-16 22:05:22.272475128 +0000 UTC m=+933.251824427"
Jul 16 22:05:23.243645 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:23.243609    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" event={"ID":"5f3ef3c9-7687-4da2-a037-a833a67c9c0d","Type":"ContainerStarted","Data":"77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d"}
Jul 16 22:05:23.271383 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:23.271324    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podStartSLOduration=4.174784172 podStartE2EDuration="7.271303501s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.096519329s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:05:23.266758005 +0000 UTC m=+934.246107306" watchObservedRunningTime="2026-07-16 22:05:23.271303501 +0000 UTC m=+934.250652803"
Jul 16 22:05:24.010744 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:24.010714    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:27.011543 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:27.011508    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:05:27.013504 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:27.013474    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:05:37.011631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:37.011586    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:05:47.011722 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:47.011676    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:05:53.245941 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:53.245914    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:05:57.011657 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:05:57.011612    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:06:07.011415 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:06:07.011364    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:06:17.011592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:06:17.011490    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:06:27.011422 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:06:27.011376    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:06:37.011988 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:06:37.011944    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:06:47.011678 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:06:47.011634    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:06:57.012034 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:06:57.011993    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:07:07.011602 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:07:07.011549    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:07:17.021000 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:07:17.020967    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:07:17.028453 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:07:17.028433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:07:56.706578 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:07:56.706549    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"]
Jul 16 22:07:56.716520 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:07:56.716493    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"]
Jul 16 22:08:03.245948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:03.245907    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:08:11.729384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:11.729328    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="main" containerID="cri-o://6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224" gracePeriod=45
Jul 16 22:08:11.743091 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:11.743030    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" containerID="cri-o://77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d" gracePeriod=45
Jul 16 22:08:11.988987 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:11.988925    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:08:12.022304 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.022259    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 16 22:08:12.126851 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.126796    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bcf8e4-9551-4999-a508-ac8d7443af71-tls-certs\") pod \"57bcf8e4-9551-4999-a508-ac8d7443af71\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") "
Jul 16 22:08:12.127025 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.126882    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-tokenizer-uds\") pod \"57bcf8e4-9551-4999-a508-ac8d7443af71\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") "
Jul 16 22:08:12.127025 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.126977    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pccqf\" (UniqueName: \"kubernetes.io/projected/57bcf8e4-9551-4999-a508-ac8d7443af71-kube-api-access-pccqf\") pod \"57bcf8e4-9551-4999-a508-ac8d7443af71\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") "
Jul 16 22:08:12.127142 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.127075    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-kserve-provision-location\") pod \"57bcf8e4-9551-4999-a508-ac8d7443af71\" (UID: \"57bcf8e4-9551-4999-a508-ac8d7443af71\") "
Jul 16 22:08:12.127142 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.127088    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "57bcf8e4-9551-4999-a508-ac8d7443af71" (UID: "57bcf8e4-9551-4999-a508-ac8d7443af71"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.127327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.127313    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-tokenizer-uds\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.127781 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.127756    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57bcf8e4-9551-4999-a508-ac8d7443af71" (UID: "57bcf8e4-9551-4999-a508-ac8d7443af71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.129132 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.129094    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bcf8e4-9551-4999-a508-ac8d7443af71-kube-api-access-pccqf" (OuterVolumeSpecName: "kube-api-access-pccqf") pod "57bcf8e4-9551-4999-a508-ac8d7443af71" (UID: "57bcf8e4-9551-4999-a508-ac8d7443af71"). InnerVolumeSpecName "kube-api-access-pccqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:08:12.129213 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.129173    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bcf8e4-9551-4999-a508-ac8d7443af71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57bcf8e4-9551-4999-a508-ac8d7443af71" (UID: "57bcf8e4-9551-4999-a508-ac8d7443af71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:08:12.191741 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.191704    2557 generic.go:356] "Generic (PLEG): container finished" podID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerID="6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224" exitCode=0
Jul 16 22:08:12.191948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.191753    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"
Jul 16 22:08:12.191948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.191781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" event={"ID":"57bcf8e4-9551-4999-a508-ac8d7443af71","Type":"ContainerDied","Data":"6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224"}
Jul 16 22:08:12.191948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.191863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm" event={"ID":"57bcf8e4-9551-4999-a508-ac8d7443af71","Type":"ContainerDied","Data":"2941ea7762ec3316e00c39e5726b6811b320de0a81e38ac1715f359abcf5a91f"}
Jul 16 22:08:12.191948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.191891    2557 scope.go:117] "RemoveContainer" containerID="6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224"
Jul 16 22:08:12.205727 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.205706    2557 scope.go:117] "RemoveContainer" containerID="a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe"
Jul 16 22:08:12.214163 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.214137    2557 scope.go:117] "RemoveContainer" containerID="6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224"
Jul 16 22:08:12.214465 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:08:12.214445    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224\": container with ID starting with 6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224 not found: ID does not exist" containerID="6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224"
Jul 16 22:08:12.214541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.214474    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224"} err="failed to get container status \"6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224\": rpc error: code = NotFound desc = could not find container \"6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224\": container with ID starting with 6b9ada9db29bf668e89e5c4afe4cfae497080f16ee6733900029980c286e1224 not found: ID does not exist"
Jul 16 22:08:12.214541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.214491    2557 scope.go:117] "RemoveContainer" containerID="a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe"
Jul 16 22:08:12.214760 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:08:12.214738    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe\": container with ID starting with a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe not found: ID does not exist" containerID="a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe"
Jul 16 22:08:12.214801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.214774    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe"} err="failed to get container status \"a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe\": rpc error: code = NotFound desc = could not find container \"a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe\": container with ID starting with a0205d07acb948b3762b023b2a0cf270276a31fbacfbd812febad23b417441fe not found: ID does not exist"
Jul 16 22:08:12.216206 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.216187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"]
Jul 16 22:08:12.219787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.219766    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-66fccd984fhr5cm"]
Jul 16 22:08:12.227738 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.227718    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bcf8e4-9551-4999-a508-ac8d7443af71-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.227804 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.227740    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pccqf\" (UniqueName: \"kubernetes.io/projected/57bcf8e4-9551-4999-a508-ac8d7443af71-kube-api-access-pccqf\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.227804 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.227750    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bcf8e4-9551-4999-a508-ac8d7443af71-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.498550 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.498520    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:08:12.631148 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tmp-dir\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631148 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631106    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tls-certs\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631148 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631129    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-dshm\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631433 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631154    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-model-cache\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631433 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631180    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvlbc\" (UniqueName: \"kubernetes.io/projected/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kube-api-access-pvlbc\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631433 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631212    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kserve-provision-location\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631433 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631242    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-home\") pod \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\" (UID: \"5f3ef3c9-7687-4da2-a037-a833a67c9c0d\") "
Jul 16 22:08:12.631652 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631499    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.631988 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.631965    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-home" (OuterVolumeSpecName: "home") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.633864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.633778    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-dshm" (OuterVolumeSpecName: "dshm") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.634014 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.633914    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kube-api-access-pvlbc" (OuterVolumeSpecName: "kube-api-access-pvlbc") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "kube-api-access-pvlbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:08:12.634014 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.633990    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:08:12.644329 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.644294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.691063 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.691022    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f3ef3c9-7687-4da2-a037-a833a67c9c0d" (UID: "5f3ef3c9-7687-4da2-a037-a833a67c9c0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:08:12.732877 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732808    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.732877 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732871    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.732877 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732884    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.733330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732895    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.733330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732907    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlbc\" (UniqueName: \"kubernetes.io/projected/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kube-api-access-pvlbc\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.733330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732919    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:12.733330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:12.732930    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3ef3c9-7687-4da2-a037-a833a67c9c0d-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:08:13.200028 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.199984    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerID="77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d" exitCode=0
Jul 16 22:08:13.200233 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.200042    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"
Jul 16 22:08:13.200233 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.200048    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" event={"ID":"5f3ef3c9-7687-4da2-a037-a833a67c9c0d","Type":"ContainerDied","Data":"77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d"}
Jul 16 22:08:13.200233 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.200108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27" event={"ID":"5f3ef3c9-7687-4da2-a037-a833a67c9c0d","Type":"ContainerDied","Data":"5023ec090168d25fe4f6e0ad26ba61e58b0f1be980ec639971d8381a7914c0f5"}
Jul 16 22:08:13.200233 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.200137    2557 scope.go:117] "RemoveContainer" containerID="77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d"
Jul 16 22:08:13.210315 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.210297    2557 scope.go:117] "RemoveContainer" containerID="1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33"
Jul 16 22:08:13.219502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.219424    2557 scope.go:117] "RemoveContainer" containerID="77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d"
Jul 16 22:08:13.219980 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:08:13.219947    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d\": container with ID starting with 77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d not found: ID does not exist" containerID="77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d"
Jul 16 22:08:13.220080 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.219993    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d"} err="failed to get container status \"77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d\": rpc error: code = NotFound desc = could not find container \"77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d\": container with ID starting with 77ae0db817d2f24343cac51f0e1f93de95658ac641655bff46cf83cf59a7b22d not found: ID does not exist"
Jul 16 22:08:13.220080 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.220019    2557 scope.go:117] "RemoveContainer" containerID="1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33"
Jul 16 22:08:13.220334 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:08:13.220311    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33\": container with ID starting with 1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33 not found: ID does not exist" containerID="1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33"
Jul 16 22:08:13.220385 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.220341    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33"} err="failed to get container status \"1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33\": rpc error: code = NotFound desc = could not find container \"1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33\": container with ID starting with 1cd3029eaf7b63e1ea00c4933d0269367f2e4803ecebe3808ba5a694d509eb33 not found: ID does not exist"
Jul 16 22:08:13.240750 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.240727    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"]
Jul 16 22:08:13.249182 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.249159    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c78d67c56-sxd27"]
Jul 16 22:08:13.568157 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.568074    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" path="/var/lib/kubelet/pods/57bcf8e4-9551-4999-a508-ac8d7443af71/volumes"
Jul 16 22:08:13.568487 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:13.568474    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" path="/var/lib/kubelet/pods/5f3ef3c9-7687-4da2-a037-a833a67c9c0d/volumes"
Jul 16 22:08:49.211387 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211351    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"]
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211731    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211743    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211758    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="main"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211766    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="main"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211774    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="storage-initializer"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211780    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="storage-initializer"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211786    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="storage-initializer"
Jul 16 22:08:49.211823 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211791    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="storage-initializer"
Jul 16 22:08:49.212076 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211934    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3ef3c9-7687-4da2-a037-a833a67c9c0d" containerName="main"
Jul 16 22:08:49.212076 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.211945    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bcf8e4-9551-4999-a508-ac8d7443af71" containerName="main"
Jul 16 22:08:49.215257 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.215241    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.217951 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.217927    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 16 22:08:49.218120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.218101    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:08:49.218220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.218200    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:08:49.218427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.218410    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:08:49.218471 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.218442    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:08:49.227289 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.227269    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"]
Jul 16 22:08:49.241320 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-home\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.241438 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241362    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kserve-provision-location\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.241507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7l6f\" (UniqueName: \"kubernetes.io/projected/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kube-api-access-v7l6f\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.241507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-model-cache\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.241507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241500    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tls-certs\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.241659 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241560    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-dshm\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.241659 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.241584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tmp-dir\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.342684 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-home\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.342869 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kserve-provision-location\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.342869 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7l6f\" (UniqueName: \"kubernetes.io/projected/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kube-api-access-v7l6f\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.342869 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-model-cache\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.342869 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tls-certs\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.343068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-dshm\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.343068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.342894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tmp-dir\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.343226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.343208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-model-cache\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.343295 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.343218    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kserve-provision-location\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.343295 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.343272    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tmp-dir\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.343395 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.343309    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-home\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.345282 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.345257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-dshm\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.345533 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.345518    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tls-certs\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.352250 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.352228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7l6f\" (UniqueName: \"kubernetes.io/projected/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kube-api-access-v7l6f\") pod \"auth-disabled-test-kserve-7998df8854-d9nzk\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.528446 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.528365    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:08:49.536380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.536357    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:49.620929 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.620897    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"]
Jul 16 22:08:49.625985 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.625949    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.628697 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.628678    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-c9m4w\""
Jul 16 22:08:49.637055 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.636984    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"]
Jul 16 22:08:49.645291 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.645268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.645412 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.645312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshdg\" (UniqueName: \"kubernetes.io/projected/3d3b30be-ad11-4074-aac9-097b09a51284-kube-api-access-kshdg\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.645491 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.645404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3b30be-ad11-4074-aac9-097b09a51284-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.645491 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.645472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.684228 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.684203    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"]
Jul 16 22:08:49.686378 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:08:49.686352    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976cc958_eaf5_41ff_9b0d_2dc86c40ddf0.slice/crio-4035a843fca041b8aeff8a568bb6aad8853bb18339ace932ffa3900a1edb31ae WatchSource:0}: Error finding container 4035a843fca041b8aeff8a568bb6aad8853bb18339ace932ffa3900a1edb31ae: Status 404 returned error can't find the container with id 4035a843fca041b8aeff8a568bb6aad8853bb18339ace932ffa3900a1edb31ae
Jul 16 22:08:49.691405 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.691385    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:08:49.746502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.746472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.746614 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.746519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.746614 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.746561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshdg\" (UniqueName: \"kubernetes.io/projected/3d3b30be-ad11-4074-aac9-097b09a51284-kube-api-access-kshdg\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.746614 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.746599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3b30be-ad11-4074-aac9-097b09a51284-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.746864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.746846    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.746950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.746931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.749039 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.749019    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3b30be-ad11-4074-aac9-097b09a51284-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.755247 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.755227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshdg\" (UniqueName: \"kubernetes.io/projected/3d3b30be-ad11-4074-aac9-097b09a51284-kube-api-access-kshdg\") pod \"auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:49.939279 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:49.939245    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:50.089967 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:50.089940    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"]
Jul 16 22:08:50.091744 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:08:50.091710    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3b30be_ad11_4074_aac9_097b09a51284.slice/crio-03c529d6a54a2c1664c5aba2628d1e509b5f1f68f2687343918613e8603fb974 WatchSource:0}: Error finding container 03c529d6a54a2c1664c5aba2628d1e509b5f1f68f2687343918613e8603fb974: Status 404 returned error can't find the container with id 03c529d6a54a2c1664c5aba2628d1e509b5f1f68f2687343918613e8603fb974
Jul 16 22:08:50.394699 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:50.394667    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" event={"ID":"3d3b30be-ad11-4074-aac9-097b09a51284","Type":"ContainerStarted","Data":"d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501"}
Jul 16 22:08:50.395207 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:50.394732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" event={"ID":"3d3b30be-ad11-4074-aac9-097b09a51284","Type":"ContainerStarted","Data":"03c529d6a54a2c1664c5aba2628d1e509b5f1f68f2687343918613e8603fb974"}
Jul 16 22:08:50.396580 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:50.396538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" event={"ID":"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0","Type":"ContainerStarted","Data":"524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b"}
Jul 16 22:08:50.396710 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:50.396594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" event={"ID":"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0","Type":"ContainerStarted","Data":"4035a843fca041b8aeff8a568bb6aad8853bb18339ace932ffa3900a1edb31ae"}
Jul 16 22:08:51.403167 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:51.403135    2557 generic.go:356] "Generic (PLEG): container finished" podID="3d3b30be-ad11-4074-aac9-097b09a51284" containerID="d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501" exitCode=0
Jul 16 22:08:51.403551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:51.403225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" event={"ID":"3d3b30be-ad11-4074-aac9-097b09a51284","Type":"ContainerDied","Data":"d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501"}
Jul 16 22:08:52.410572 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:52.410536    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" event={"ID":"3d3b30be-ad11-4074-aac9-097b09a51284","Type":"ContainerStarted","Data":"c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c"}
Jul 16 22:08:52.411018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:52.410673    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:08:52.431226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:52.431147    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" podStartSLOduration=2.618379711 podStartE2EDuration="3.431123861s" totalImagesPullingTime="0s" totalInitContainerRuntime="812.74415ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:52.429165371 +0000 UTC m=+1143.408514670" watchObservedRunningTime="2026-07-16 22:08:52.431123861 +0000 UTC m=+1143.410473161"
Jul 16 22:08:53.419288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:53.419254    2557 generic.go:356] "Generic (PLEG): container finished" podID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerID="524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b" exitCode=0
Jul 16 22:08:53.419770 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:53.419346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" event={"ID":"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0","Type":"ContainerDied","Data":"524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b"}
Jul 16 22:08:54.427194 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:54.427162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" event={"ID":"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0","Type":"ContainerStarted","Data":"ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e"}
Jul 16 22:08:54.449790 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:54.449730    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podStartSLOduration=2.425562076 podStartE2EDuration="5.449689248s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.024127172s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:54.448521155 +0000 UTC m=+1145.427870454" watchObservedRunningTime="2026-07-16 22:08:54.449689248 +0000 UTC m=+1145.429038547"
Jul 16 22:08:54.536586 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:54.536542    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:59.537540 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:59.537504    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:08:59.539589 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:08:59.539558    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:09:09.537135 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:09.537084    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:09:19.536998 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:19.536898    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:09:23.421320 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:23.421291    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:09:29.537376 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:29.537329    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:09:39.537103 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:39.537058    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:09:49.536890 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:49.536846    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:09:49.545981 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:49.545954    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:09:49.546224 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:49.546205    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:09:49.550737 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:49.550709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:09:49.550875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:49.550712    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:09:59.537094 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:09:59.537043    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:10:02.218701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.218668    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"]
Jul 16 22:10:02.221520 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.221499    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.224622 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.224602    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:10:02.224734 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.224674    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 16 22:10:02.238521 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.238503    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"]
Jul 16 22:10:02.293059 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293207 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293207 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293282 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293234    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293282 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5d5m\" (UniqueName: \"kubernetes.io/projected/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kube-api-access-p5d5m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293353 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293305    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293353 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.293416 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.293364    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.393868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.393826    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.393868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.393871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.393915    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.393949    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.393968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5d5m\" (UniqueName: \"kubernetes.io/projected/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kube-api-access-p5d5m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.393991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.394019    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.394044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.394362    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.394380    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.394514    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.394638 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.394614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.395125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.395101    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.396459 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.396437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.396655 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.396635    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.402531 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.402508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5d5m\" (UniqueName: \"kubernetes.io/projected/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kube-api-access-p5d5m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.532154 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.532068    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:02.737330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.737277    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"]
Jul 16 22:10:02.740780 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:10:02.740748    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838bbc1a_4e43_4e9a_9024_ba668bfb5f0c.slice/crio-cddd0a805db234803e18205ddea79347ceb5c4c715a789237da14c698ddaa86d WatchSource:0}: Error finding container cddd0a805db234803e18205ddea79347ceb5c4c715a789237da14c698ddaa86d: Status 404 returned error can't find the container with id cddd0a805db234803e18205ddea79347ceb5c4c715a789237da14c698ddaa86d
Jul 16 22:10:02.821051 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.821017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" event={"ID":"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c","Type":"ContainerStarted","Data":"86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962"}
Jul 16 22:10:02.821176 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:02.821077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" event={"ID":"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c","Type":"ContainerStarted","Data":"cddd0a805db234803e18205ddea79347ceb5c4c715a789237da14c698ddaa86d"}
Jul 16 22:10:06.849912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:06.849871    2557 generic.go:356] "Generic (PLEG): container finished" podID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerID="86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962" exitCode=0
Jul 16 22:10:06.850389 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:06.849943    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" event={"ID":"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c","Type":"ContainerDied","Data":"86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962"}
Jul 16 22:10:07.859746 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:07.859712    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" event={"ID":"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c","Type":"ContainerStarted","Data":"7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc"}
Jul 16 22:10:07.884760 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:07.884698    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podStartSLOduration=2.7796131060000002 podStartE2EDuration="5.884678357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.105065251s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:10:07.882105588 +0000 UTC m=+1218.861454899" watchObservedRunningTime="2026-07-16 22:10:07.884678357 +0000 UTC m=+1218.864027656"
Jul 16 22:10:08.532905 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:08.532869    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:09.537208 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:09.537158    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:10:12.533094 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:12.533044    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:10:12.535134 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:12.535103    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:10:19.537325 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:19.537274    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:10:22.533460 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:22.533409    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:10:29.537373 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:29.537325    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:10:32.533364 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:32.533322    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:10:39.537067 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:39.537028    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:10:42.533118 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:42.533029    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:10:49.547180 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:49.547152    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:10:49.555643 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:49.555614    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:10:52.532953 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:52.532905    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:10:54.315296 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:54.315261    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"]
Jul 16 22:10:54.320698 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:10:54.320666    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"]
Jul 16 22:11:02.532607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:02.532553    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:11:03.421884 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:03.421842    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:11:09.346088 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.346003    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="main" containerID="cri-o://c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c" gracePeriod=45
Jul 16 22:11:09.354250 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.354214    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" containerID="cri-o://ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e" gracePeriod=45
Jul 16 22:11:09.548225 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.548194    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 16 22:11:09.622477 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.622451    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:11:09.734668 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.734636    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-kserve-provision-location\") pod \"3d3b30be-ad11-4074-aac9-097b09a51284\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") "
Jul 16 22:11:09.734878 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.734679    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kshdg\" (UniqueName: \"kubernetes.io/projected/3d3b30be-ad11-4074-aac9-097b09a51284-kube-api-access-kshdg\") pod \"3d3b30be-ad11-4074-aac9-097b09a51284\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") "
Jul 16 22:11:09.734878 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.734711    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-tokenizer-uds\") pod \"3d3b30be-ad11-4074-aac9-097b09a51284\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") "
Jul 16 22:11:09.734878 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.734755    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3b30be-ad11-4074-aac9-097b09a51284-tls-certs\") pod \"3d3b30be-ad11-4074-aac9-097b09a51284\" (UID: \"3d3b30be-ad11-4074-aac9-097b09a51284\") "
Jul 16 22:11:09.735066 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.735008    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3d3b30be-ad11-4074-aac9-097b09a51284" (UID: "3d3b30be-ad11-4074-aac9-097b09a51284"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:09.735492 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.735420    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d3b30be-ad11-4074-aac9-097b09a51284" (UID: "3d3b30be-ad11-4074-aac9-097b09a51284"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:09.736993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.736964    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3b30be-ad11-4074-aac9-097b09a51284-kube-api-access-kshdg" (OuterVolumeSpecName: "kube-api-access-kshdg") pod "3d3b30be-ad11-4074-aac9-097b09a51284" (UID: "3d3b30be-ad11-4074-aac9-097b09a51284"). InnerVolumeSpecName "kube-api-access-kshdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:11:09.736993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.736982    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3b30be-ad11-4074-aac9-097b09a51284-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d3b30be-ad11-4074-aac9-097b09a51284" (UID: "3d3b30be-ad11-4074-aac9-097b09a51284"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:11:09.836011 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.835976    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-tokenizer-uds\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:09.836011 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.836010    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3b30be-ad11-4074-aac9-097b09a51284-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:09.836241 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.836022    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d3b30be-ad11-4074-aac9-097b09a51284-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:09.836241 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:09.836035    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshdg\" (UniqueName: \"kubernetes.io/projected/3d3b30be-ad11-4074-aac9-097b09a51284-kube-api-access-kshdg\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:09.938395 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:11:09.938366    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976cc958_eaf5_41ff_9b0d_2dc86c40ddf0.slice/crio-conmon-ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:11:10.118487 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.118458    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:11:10.239071 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239034    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7l6f\" (UniqueName: \"kubernetes.io/projected/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kube-api-access-v7l6f\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239071 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239077    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-dshm\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239264    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-home\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239345 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239307    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tmp-dir\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239405 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239367    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kserve-provision-location\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239405 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239393    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tls-certs\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239539 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239431    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-model-cache\") pod \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\" (UID: \"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0\") "
Jul 16 22:11:10.239944 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.239915    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:10.240236 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.240204    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-home" (OuterVolumeSpecName: "home") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:10.241427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.241401    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-dshm" (OuterVolumeSpecName: "dshm") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:10.241637 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.241622    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kube-api-access-v7l6f" (OuterVolumeSpecName: "kube-api-access-v7l6f") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "kube-api-access-v7l6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:11:10.241863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.241835    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:11:10.258551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.258525    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:10.261043 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.260998    2557 generic.go:356] "Generic (PLEG): container finished" podID="3d3b30be-ad11-4074-aac9-097b09a51284" containerID="c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c" exitCode=0
Jul 16 22:11:10.261149 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.261080    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" event={"ID":"3d3b30be-ad11-4074-aac9-097b09a51284","Type":"ContainerDied","Data":"c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c"}
Jul 16 22:11:10.261149 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.261138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn" event={"ID":"3d3b30be-ad11-4074-aac9-097b09a51284","Type":"ContainerDied","Data":"03c529d6a54a2c1664c5aba2628d1e509b5f1f68f2687343918613e8603fb974"}
Jul 16 22:11:10.261287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.261165    2557 scope.go:117] "RemoveContainer" containerID="c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c"
Jul 16 22:11:10.261287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.261197    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"
Jul 16 22:11:10.263144 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.263118    2557 generic.go:356] "Generic (PLEG): container finished" podID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerID="ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e" exitCode=0
Jul 16 22:11:10.263233 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.263200    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" event={"ID":"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0","Type":"ContainerDied","Data":"ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e"}
Jul 16 22:11:10.263306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.263239    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk" event={"ID":"976cc958-eaf5-41ff-9b0d-2dc86c40ddf0","Type":"ContainerDied","Data":"4035a843fca041b8aeff8a568bb6aad8853bb18339ace932ffa3900a1edb31ae"}
Jul 16 22:11:10.263366 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.263309    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"
Jul 16 22:11:10.272112 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.272089    2557 scope.go:117] "RemoveContainer" containerID="d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501"
Jul 16 22:11:10.282917 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.282898    2557 scope.go:117] "RemoveContainer" containerID="c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c"
Jul 16 22:11:10.283215 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:11:10.283195    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c\": container with ID starting with c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c not found: ID does not exist" containerID="c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c"
Jul 16 22:11:10.283327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.283224    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c"} err="failed to get container status \"c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c\": rpc error: code = NotFound desc = could not find container \"c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c\": container with ID starting with c8602a2fef638aa8853b18d87cde6995f2d2777872de0e48a32082cf8f7acc8c not found: ID does not exist"
Jul 16 22:11:10.283327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.283245    2557 scope.go:117] "RemoveContainer" containerID="d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501"
Jul 16 22:11:10.283472 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:11:10.283454    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501\": container with ID starting with d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501 not found: ID does not exist" containerID="d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501"
Jul 16 22:11:10.283512 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.283478    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501"} err="failed to get container status \"d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501\": rpc error: code = NotFound desc = could not find container \"d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501\": container with ID starting with d21b0b406b76b89d9c3470dfcfb7fb52647b5973470fffa129306cb169420501 not found: ID does not exist"
Jul 16 22:11:10.283512 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.283495    2557 scope.go:117] "RemoveContainer" containerID="ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e"
Jul 16 22:11:10.293099 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.293071    2557 scope.go:117] "RemoveContainer" containerID="524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b"
Jul 16 22:11:10.300558 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.300536    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"]
Jul 16 22:11:10.303168 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.303153    2557 scope.go:117] "RemoveContainer" containerID="ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e"
Jul 16 22:11:10.303481 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:11:10.303459    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e\": container with ID starting with ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e not found: ID does not exist" containerID="ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e"
Jul 16 22:11:10.303553 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.303493    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e"} err="failed to get container status \"ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e\": rpc error: code = NotFound desc = could not find container \"ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e\": container with ID starting with ce64bd25252d82c37511c8b78e2092a0eeadf34f9349b17a586c669e7627cc7e not found: ID does not exist"
Jul 16 22:11:10.303553 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.303517    2557 scope.go:117] "RemoveContainer" containerID="524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b"
Jul 16 22:11:10.303788 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:11:10.303760    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b\": container with ID starting with 524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b not found: ID does not exist" containerID="524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b"
Jul 16 22:11:10.303891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.303788    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b"} err="failed to get container status \"524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b\": rpc error: code = NotFound desc = could not find container \"524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b\": container with ID starting with 524e9b4157bf6963a0920062a29987537a58b66aaa38c6b4309474214e22424b not found: ID does not exist"
Jul 16 22:11:10.310050 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.310031    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cbd548bb4-6hsgn"]
Jul 16 22:11:10.317548 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.317520    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" (UID: "976cc958-eaf5-41ff-9b0d-2dc86c40ddf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:11:10.340863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340832    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7l6f\" (UniqueName: \"kubernetes.io/projected/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kube-api-access-v7l6f\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.340863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340857    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.340863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340866    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.341070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340896    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.341070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340905    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.341070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340914    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.341070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.340923    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:11:10.587582 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.587545    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"]
Jul 16 22:11:10.590889 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:10.590855    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7998df8854-d9nzk"]
Jul 16 22:11:11.564180 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:11.564142    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" path="/var/lib/kubelet/pods/3d3b30be-ad11-4074-aac9-097b09a51284/volumes"
Jul 16 22:11:11.564596 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:11.564583    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" path="/var/lib/kubelet/pods/976cc958-eaf5-41ff-9b0d-2dc86c40ddf0/volumes"
Jul 16 22:11:12.533291 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:12.533248    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 16 22:11:22.543149 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:22.543114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:11:22.551459 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:11:22.551427    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:12:02.210398 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.210354    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"]
Jul 16 22:12:02.211009 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.210989    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="storage-initializer"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211013    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="storage-initializer"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211032    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="main"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211040    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="main"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211050    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211058    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211067    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="storage-initializer"
Jul 16 22:12:02.211093 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211076    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="storage-initializer"
Jul 16 22:12:02.211435 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211170    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3b30be-ad11-4074-aac9-097b09a51284" containerName="main"
Jul 16 22:12:02.211435 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.211185    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="976cc958-eaf5-41ff-9b0d-2dc86c40ddf0" containerName="main"
Jul 16 22:12:02.214394 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.214374    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.217063 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.217039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 16 22:12:02.228961 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.228936    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"]
Jul 16 22:12:02.313303 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.313303 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.313552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmdz\" (UniqueName: \"kubernetes.io/projected/3e7bef2b-28fb-4568-adac-6df2b4adc720-kube-api-access-5zmdz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.313552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.313552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313466    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.313552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.313552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.313512    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e7bef2b-28fb-4568-adac-6df2b4adc720-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414071    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414168    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmdz\" (UniqueName: \"kubernetes.io/projected/3e7bef2b-28fb-4568-adac-6df2b4adc720-kube-api-access-5zmdz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414235    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e7bef2b-28fb-4568-adac-6df2b4adc720-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414619    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.414669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.414641    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.416686 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.416665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.416989 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.416969    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e7bef2b-28fb-4568-adac-6df2b4adc720-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.422963 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.422941    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmdz\" (UniqueName: \"kubernetes.io/projected/3e7bef2b-28fb-4568-adac-6df2b4adc720-kube-api-access-5zmdz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.524777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.524678    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:02.678675 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:02.678584    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"]
Jul 16 22:12:02.680980 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:12:02.680953    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7bef2b_28fb_4568_adac_6df2b4adc720.slice/crio-ba1108951f0050639985765810d89378d557ba90797a5fe89a28e89f0f2ccf5a WatchSource:0}: Error finding container ba1108951f0050639985765810d89378d557ba90797a5fe89a28e89f0f2ccf5a: Status 404 returned error can't find the container with id ba1108951f0050639985765810d89378d557ba90797a5fe89a28e89f0f2ccf5a
Jul 16 22:12:03.573224 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:03.573185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" event={"ID":"3e7bef2b-28fb-4568-adac-6df2b4adc720","Type":"ContainerStarted","Data":"c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f"}
Jul 16 22:12:03.573648 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:03.573248    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" event={"ID":"3e7bef2b-28fb-4568-adac-6df2b4adc720","Type":"ContainerStarted","Data":"ba1108951f0050639985765810d89378d557ba90797a5fe89a28e89f0f2ccf5a"}
Jul 16 22:12:05.467874 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:05.467841    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"]
Jul 16 22:12:06.599311 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:06.599276    2557 generic.go:356] "Generic (PLEG): container finished" podID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerID="c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f" exitCode=0
Jul 16 22:12:06.599704 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:06.599327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" event={"ID":"3e7bef2b-28fb-4568-adac-6df2b4adc720","Type":"ContainerDied","Data":"c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f"}
Jul 16 22:12:07.607580 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:07.607532    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" event={"ID":"3e7bef2b-28fb-4568-adac-6df2b4adc720","Type":"ContainerStarted","Data":"b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f"}
Jul 16 22:12:07.630266 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:07.630208    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podStartSLOduration=2.625142233 podStartE2EDuration="5.630190137s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.005047904s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:07.628159501 +0000 UTC m=+1338.607508812" watchObservedRunningTime="2026-07-16 22:12:07.630190137 +0000 UTC m=+1338.609539437"
Jul 16 22:12:08.525468 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:08.525428    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:12.525833 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:12.525710    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:12:12.527444 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:12.527415    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:12:20.495790 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:20.495723    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main" containerID="cri-o://7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc" gracePeriod=45
Jul 16 22:12:21.159614 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.159583    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:12:21.296096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.295995    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kserve-provision-location\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296039    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5d5m\" (UniqueName: \"kubernetes.io/projected/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kube-api-access-p5d5m\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296068    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-model-cache\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296141    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tmp-dir\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296179    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-dshm\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296200    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-home\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296260    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tls-certs\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296318    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-cabundle-cert\") pod \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\" (UID: \"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c\") "
Jul 16 22:12:21.296654 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296404    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:21.296654 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296616    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.296776 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296621    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-home" (OuterVolumeSpecName: "home") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:21.296894 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.296867    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:21.298647 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.298610    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kube-api-access-p5d5m" (OuterVolumeSpecName: "kube-api-access-p5d5m") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "kube-api-access-p5d5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:21.298987 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.298959    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:21.299144 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.299124    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-dshm" (OuterVolumeSpecName: "dshm") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:21.311073 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.311011    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:21.353369 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.353321    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" (UID: "838bbc1a-4e43-4e9a-9024-ba668bfb5f0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:21.397694 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397655    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.397694 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397684    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.397694 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397694    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.397694 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397701    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.398084 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397710    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-cabundle-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.398084 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397719    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.398084 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.397727    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5d5m\" (UniqueName: \"kubernetes.io/projected/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c-kube-api-access-p5d5m\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.690146 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.690111    2557 generic.go:356] "Generic (PLEG): container finished" podID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerID="7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc" exitCode=0
Jul 16 22:12:21.690633 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.690161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" event={"ID":"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c","Type":"ContainerDied","Data":"7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc"}
Jul 16 22:12:21.690633 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.690177    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"
Jul 16 22:12:21.690633 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.690220    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf" event={"ID":"838bbc1a-4e43-4e9a-9024-ba668bfb5f0c","Type":"ContainerDied","Data":"cddd0a805db234803e18205ddea79347ceb5c4c715a789237da14c698ddaa86d"}
Jul 16 22:12:21.690633 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.690243    2557 scope.go:117] "RemoveContainer" containerID="7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc"
Jul 16 22:12:21.700013 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.699997    2557 scope.go:117] "RemoveContainer" containerID="86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962"
Jul 16 22:12:21.709928 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.709908    2557 scope.go:117] "RemoveContainer" containerID="7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc"
Jul 16 22:12:21.710226 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:12:21.710205    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc\": container with ID starting with 7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc not found: ID does not exist" containerID="7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc"
Jul 16 22:12:21.710306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.710238    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc"} err="failed to get container status \"7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc\": rpc error: code = NotFound desc = could not find container \"7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc\": container with ID starting with 7eb3af4a27e6a9f09a201e45c0d534f98551c17e2e2b68122e4c75011f77aebc not found: ID does not exist"
Jul 16 22:12:21.710306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.710262    2557 scope.go:117] "RemoveContainer" containerID="86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962"
Jul 16 22:12:21.710565 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:12:21.710542    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962\": container with ID starting with 86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962 not found: ID does not exist" containerID="86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962"
Jul 16 22:12:21.710618 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.710572    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962"} err="failed to get container status \"86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962\": rpc error: code = NotFound desc = could not find container \"86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962\": container with ID starting with 86dfa6879b745392f4c096c6f5588e166f2608a0608fb5d4b39a73396fb63962 not found: ID does not exist"
Jul 16 22:12:21.710807 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.710792    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"]
Jul 16 22:12:21.714204 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:21.714186    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-d66767cfc-twflf"]
Jul 16 22:12:22.525322 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:22.525286    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:12:23.562386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:23.562349    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" path="/var/lib/kubelet/pods/838bbc1a-4e43-4e9a-9024-ba668bfb5f0c/volumes"
Jul 16 22:12:32.525589 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:32.525542    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:12:36.317351 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.317315    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"]
Jul 16 22:12:36.318276 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.318247    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="storage-initializer"
Jul 16 22:12:36.318276 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.318271    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="storage-initializer"
Jul 16 22:12:36.318426 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.318301    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main"
Jul 16 22:12:36.318426 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.318307    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main"
Jul 16 22:12:36.318426 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.318360    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="838bbc1a-4e43-4e9a-9024-ba668bfb5f0c" containerName="main"
Jul 16 22:12:36.321125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.321108    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.324101 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.324077    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:12:36.324205 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.324118    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 16 22:12:36.339327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.339306    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"]
Jul 16 22:12:36.430609 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430609 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb508b7-deb9-47ec-876d-8e835a2abefa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430803 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430655    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j658\" (UniqueName: \"kubernetes.io/projected/4cb508b7-deb9-47ec-876d-8e835a2abefa-kube-api-access-7j658\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430803 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430726    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cb508b7-deb9-47ec-876d-8e835a2abefa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430803 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430929 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430929 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.430929 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.430884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532029 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.531993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532029 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532035    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532256 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532256 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb508b7-deb9-47ec-876d-8e835a2abefa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532256 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j658\" (UniqueName: \"kubernetes.io/projected/4cb508b7-deb9-47ec-876d-8e835a2abefa-kube-api-access-7j658\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532256 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cb508b7-deb9-47ec-876d-8e835a2abefa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532256 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532521 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532521 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532486    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532607    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.532738 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532652    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.533020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.532994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cb508b7-deb9-47ec-876d-8e835a2abefa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.534696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.534667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.535111 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.535091    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb508b7-deb9-47ec-876d-8e835a2abefa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.541363 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.541342    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j658\" (UniqueName: \"kubernetes.io/projected/4cb508b7-deb9-47ec-876d-8e835a2abefa-kube-api-access-7j658\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.632141 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.632063    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:36.788979 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:36.788952    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"]
Jul 16 22:12:36.791714 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:12:36.791688    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb508b7_deb9_47ec_876d_8e835a2abefa.slice/crio-0727167432d47bc2ac95f5808a0cc43705dbcc7ce81fb3c0ce734f2807f10d79 WatchSource:0}: Error finding container 0727167432d47bc2ac95f5808a0cc43705dbcc7ce81fb3c0ce734f2807f10d79: Status 404 returned error can't find the container with id 0727167432d47bc2ac95f5808a0cc43705dbcc7ce81fb3c0ce734f2807f10d79
Jul 16 22:12:37.782315 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:37.782266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" event={"ID":"4cb508b7-deb9-47ec-876d-8e835a2abefa","Type":"ContainerStarted","Data":"0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea"}
Jul 16 22:12:37.782805 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:37.782421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" event={"ID":"4cb508b7-deb9-47ec-876d-8e835a2abefa","Type":"ContainerStarted","Data":"0727167432d47bc2ac95f5808a0cc43705dbcc7ce81fb3c0ce734f2807f10d79"}
Jul 16 22:12:40.803203 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:40.803166    2557 generic.go:356] "Generic (PLEG): container finished" podID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerID="0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea" exitCode=0
Jul 16 22:12:40.803570 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:40.803241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" event={"ID":"4cb508b7-deb9-47ec-876d-8e835a2abefa","Type":"ContainerDied","Data":"0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea"}
Jul 16 22:12:41.812224 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:41.812195    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" event={"ID":"4cb508b7-deb9-47ec-876d-8e835a2abefa","Type":"ContainerStarted","Data":"029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e"}
Jul 16 22:12:41.835985 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:41.835936    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podStartSLOduration=2.641467892 podStartE2EDuration="5.835918574s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.194450682s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:41.834504401 +0000 UTC m=+1372.813853700" watchObservedRunningTime="2026-07-16 22:12:41.835918574 +0000 UTC m=+1372.815267887"
Jul 16 22:12:42.525802 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:42.525753    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:12:42.632408 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:42.632376    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:46.632867 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:46.632824    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:12:46.634729 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:46.634684    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:12:52.525665 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:52.525614    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:12:56.633306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:12:56.633262    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:13:02.525582 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:02.525532    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:13:06.633373 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:06.633315    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:13:12.525445 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:12.525400    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:13:16.632760 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:16.632705    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:13:22.525350 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:22.525306    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:13:26.632990 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:26.632941    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:13:32.525940 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:32.525898    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:13:36.633363 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:36.633319    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:13:42.525519 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:42.525408    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:13:46.632625 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:46.632573    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:13:52.525184 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:52.525130    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:13:56.652684 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:56.652647    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:13:56.663787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:13:56.663754    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:14:02.534905 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:02.534875    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:14:02.542459 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:02.542429    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:14:07.989038 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:07.989005    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"]
Jul 16 22:14:10.448972 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:10.448942    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"]
Jul 16 22:14:23.012201 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.012163    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" containerID="cri-o://b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f" gracePeriod=45
Jul 16 22:14:23.535807 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.535770    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 16 22:14:23.688327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.688303    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:14:23.736647 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736623    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e7bef2b-28fb-4568-adac-6df2b4adc720-tls-certs\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.736791 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736668    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-model-cache\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.736791 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736695    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zmdz\" (UniqueName: \"kubernetes.io/projected/3e7bef2b-28fb-4568-adac-6df2b4adc720-kube-api-access-5zmdz\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.736791 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736755    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-tmp-dir\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.736982 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736800    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-dshm\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.736982 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736867    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-kserve-provision-location\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.736982 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.736982 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.736919    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-home\") pod \"3e7bef2b-28fb-4568-adac-6df2b4adc720\" (UID: \"3e7bef2b-28fb-4568-adac-6df2b4adc720\") "
Jul 16 22:14:23.737201 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.737177    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.737744 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.737690    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-home" (OuterVolumeSpecName: "home") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.739077 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.739050    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-dshm" (OuterVolumeSpecName: "dshm") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.739603 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.739577    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7bef2b-28fb-4568-adac-6df2b4adc720-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:14:23.739603 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.739591    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7bef2b-28fb-4568-adac-6df2b4adc720-kube-api-access-5zmdz" (OuterVolumeSpecName: "kube-api-access-5zmdz") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "kube-api-access-5zmdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:14:23.749051 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.749018    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.793941 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.793878    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e7bef2b-28fb-4568-adac-6df2b4adc720" (UID: "3e7bef2b-28fb-4568-adac-6df2b4adc720"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.838595 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.838571    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.838595 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.838593    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.838716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.838602    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.838716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.838610    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e7bef2b-28fb-4568-adac-6df2b4adc720-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.838716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.838618    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e7bef2b-28fb-4568-adac-6df2b4adc720-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.838716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:23.838626    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmdz\" (UniqueName: \"kubernetes.io/projected/3e7bef2b-28fb-4568-adac-6df2b4adc720-kube-api-access-5zmdz\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:24.433885 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.433842    2557 generic.go:356] "Generic (PLEG): container finished" podID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerID="b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f" exitCode=0
Jul 16 22:14:24.433885 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.433884    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"
Jul 16 22:14:24.434407 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.433902    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" event={"ID":"3e7bef2b-28fb-4568-adac-6df2b4adc720","Type":"ContainerDied","Data":"b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f"}
Jul 16 22:14:24.434407 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.433968    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4" event={"ID":"3e7bef2b-28fb-4568-adac-6df2b4adc720","Type":"ContainerDied","Data":"ba1108951f0050639985765810d89378d557ba90797a5fe89a28e89f0f2ccf5a"}
Jul 16 22:14:24.434407 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.434005    2557 scope.go:117] "RemoveContainer" containerID="b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f"
Jul 16 22:14:24.443329 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.443315    2557 scope.go:117] "RemoveContainer" containerID="c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f"
Jul 16 22:14:24.452108 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.452093    2557 scope.go:117] "RemoveContainer" containerID="b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f"
Jul 16 22:14:24.452337 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:14:24.452322    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f\": container with ID starting with b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f not found: ID does not exist" containerID="b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f"
Jul 16 22:14:24.452383 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.452344    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f"} err="failed to get container status \"b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f\": rpc error: code = NotFound desc = could not find container \"b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f\": container with ID starting with b12a43d9be832ea05975eed0a655cac3858166a22ba956b8c80c1c9b7559b93f not found: ID does not exist"
Jul 16 22:14:24.452383 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.452357    2557 scope.go:117] "RemoveContainer" containerID="c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f"
Jul 16 22:14:24.452560 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:14:24.452545    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f\": container with ID starting with c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f not found: ID does not exist" containerID="c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f"
Jul 16 22:14:24.452599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.452564    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f"} err="failed to get container status \"c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f\": rpc error: code = NotFound desc = could not find container \"c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f\": container with ID starting with c73f69c59c976b7cc2e03fc37ceac1807d77788f538d1cd648502268a69b318f not found: ID does not exist"
Jul 16 22:14:24.456411 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.456389    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"]
Jul 16 22:14:24.463607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:24.463588    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5fcd5db4ddxt8b4"]
Jul 16 22:14:25.472756 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:25.472721    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" containerID="cri-o://029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e" gracePeriod=45
Jul 16 22:14:25.563555 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:25.563520    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" path="/var/lib/kubelet/pods/3e7bef2b-28fb-4568-adac-6df2b4adc720/volumes"
Jul 16 22:14:25.653369 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:25.653334    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 16 22:14:26.023743 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.023722    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:14:26.159145 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159122    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-home\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159311 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159157    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb508b7-deb9-47ec-876d-8e835a2abefa-tls-certs\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159311 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159174    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cb508b7-deb9-47ec-876d-8e835a2abefa-cabundle-cert\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159311 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159189    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-dshm\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159311 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-kserve-provision-location\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159311 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159257    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j658\" (UniqueName: \"kubernetes.io/projected/4cb508b7-deb9-47ec-876d-8e835a2abefa-kube-api-access-7j658\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159575 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-tmp-dir\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159575 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159424    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-model-cache\") pod \"4cb508b7-deb9-47ec-876d-8e835a2abefa\" (UID: \"4cb508b7-deb9-47ec-876d-8e835a2abefa\") "
Jul 16 22:14:26.159575 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb508b7-deb9-47ec-876d-8e835a2abefa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:14:26.159575 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159557    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-home" (OuterVolumeSpecName: "home") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:26.159885 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159806    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:26.159972 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159898    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.159972 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.159916    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cb508b7-deb9-47ec-876d-8e835a2abefa-cabundle-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.160327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.160300    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:26.161441 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.161415    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb508b7-deb9-47ec-876d-8e835a2abefa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:14:26.161631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.161612    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb508b7-deb9-47ec-876d-8e835a2abefa-kube-api-access-7j658" (OuterVolumeSpecName: "kube-api-access-7j658") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "kube-api-access-7j658". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:14:26.161697 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.161683    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-dshm" (OuterVolumeSpecName: "dshm") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:26.225738 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.225704    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cb508b7-deb9-47ec-876d-8e835a2abefa" (UID: "4cb508b7-deb9-47ec-876d-8e835a2abefa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:26.261262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.261217    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb508b7-deb9-47ec-876d-8e835a2abefa-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.261262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.261259    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.261386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.261268    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.261386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.261277    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j658\" (UniqueName: \"kubernetes.io/projected/4cb508b7-deb9-47ec-876d-8e835a2abefa-kube-api-access-7j658\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.261386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.261285    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.261386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.261293    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb508b7-deb9-47ec-876d-8e835a2abefa-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.447273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.447201    2557 generic.go:356] "Generic (PLEG): container finished" podID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerID="029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e" exitCode=0
Jul 16 22:14:26.447273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.447257    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"
Jul 16 22:14:26.447427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.447296    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" event={"ID":"4cb508b7-deb9-47ec-876d-8e835a2abefa","Type":"ContainerDied","Data":"029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e"}
Jul 16 22:14:26.447427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.447355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb" event={"ID":"4cb508b7-deb9-47ec-876d-8e835a2abefa","Type":"ContainerDied","Data":"0727167432d47bc2ac95f5808a0cc43705dbcc7ce81fb3c0ce734f2807f10d79"}
Jul 16 22:14:26.447427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.447378    2557 scope.go:117] "RemoveContainer" containerID="029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e"
Jul 16 22:14:26.457178 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.457161    2557 scope.go:117] "RemoveContainer" containerID="0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea"
Jul 16 22:14:26.465975 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.465962    2557 scope.go:117] "RemoveContainer" containerID="029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e"
Jul 16 22:14:26.466185 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:14:26.466169    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e\": container with ID starting with 029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e not found: ID does not exist" containerID="029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e"
Jul 16 22:14:26.466243 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.466191    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e"} err="failed to get container status \"029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e\": rpc error: code = NotFound desc = could not find container \"029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e\": container with ID starting with 029533caf8cf17d11b91a2f2003b15e0f7f1901a1f9881ddf2992472f4864e2e not found: ID does not exist"
Jul 16 22:14:26.466243 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.466205    2557 scope.go:117] "RemoveContainer" containerID="0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea"
Jul 16 22:14:26.466425 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:14:26.466412    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea\": container with ID starting with 0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea not found: ID does not exist" containerID="0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea"
Jul 16 22:14:26.466464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.466428    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea"} err="failed to get container status \"0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea\": rpc error: code = NotFound desc = could not find container \"0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea\": container with ID starting with 0553ca7ac077f49b91afc3f12a10f9fd977bff00ceea756ae8b96532e26a16ea not found: ID does not exist"
Jul 16 22:14:26.470734 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.470711    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"]
Jul 16 22:14:26.475072 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:26.475051    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6cc48f668b-md6gb"]
Jul 16 22:14:27.567696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:27.567660    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" path="/var/lib/kubelet/pods/4cb508b7-deb9-47ec-876d-8e835a2abefa/volumes"
Jul 16 22:14:36.680676 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.680639    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"]
Jul 16 22:14:36.681170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681103    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="storage-initializer"
Jul 16 22:14:36.681170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681120    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="storage-initializer"
Jul 16 22:14:36.681170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681130    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="storage-initializer"
Jul 16 22:14:36.681170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681138    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="storage-initializer"
Jul 16 22:14:36.681170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681161    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main"
Jul 16 22:14:36.681170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681169    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main"
Jul 16 22:14:36.681511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681178    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main"
Jul 16 22:14:36.681511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681193    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main"
Jul 16 22:14:36.681511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681322    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb508b7-deb9-47ec-876d-8e835a2abefa" containerName="main"
Jul 16 22:14:36.681511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.681337    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7bef2b-28fb-4568-adac-6df2b4adc720" containerName="main"
Jul 16 22:14:36.686107 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.686090    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.688782 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.688761    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:14:36.690027 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.690005    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 16 22:14:36.690596 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.690261    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:14:36.690596 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.690333    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:14:36.690596 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.690346    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:14:36.698238 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.698218    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"]
Jul 16 22:14:36.748607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tmp-dir\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.748607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwq8h\" (UniqueName: \"kubernetes.io/projected/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kube-api-access-qwq8h\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.748900 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748672    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-model-cache\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.748900 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tls-certs\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.748900 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.748900 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-home\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.748900 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.748881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-dshm\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849690 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-model-cache\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849690 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tls-certs\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849715    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-home\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-dshm\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tmp-dir\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.849965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.849901    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwq8h\" (UniqueName: \"kubernetes.io/projected/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kube-api-access-qwq8h\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.850212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.850106    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-model-cache\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.850212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.850151    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-home\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.850212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.850170    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.850400 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.850236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tmp-dir\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.852357 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.852329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-dshm\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.852598 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.852580    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tls-certs\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.859250 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.859225    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwq8h\" (UniqueName: \"kubernetes.io/projected/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kube-api-access-qwq8h\") pod \"custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:36.996491 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:36.996400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:37.145972 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:37.145941    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"]
Jul 16 22:14:37.147180 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:14:37.147151    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36a40b2_9a92_41cc_ba6b_15a84f6b520e.slice/crio-94e56dd7b0ce252c756784b7820edcbd94c39c0cd1aaf5cda286d24a3671cd40 WatchSource:0}: Error finding container 94e56dd7b0ce252c756784b7820edcbd94c39c0cd1aaf5cda286d24a3671cd40: Status 404 returned error can't find the container with id 94e56dd7b0ce252c756784b7820edcbd94c39c0cd1aaf5cda286d24a3671cd40
Jul 16 22:14:37.506308 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:37.506264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" event={"ID":"c36a40b2-9a92-41cc-ba6b-15a84f6b520e","Type":"ContainerStarted","Data":"0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829"}
Jul 16 22:14:37.506529 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:37.506342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" event={"ID":"c36a40b2-9a92-41cc-ba6b-15a84f6b520e","Type":"ContainerStarted","Data":"94e56dd7b0ce252c756784b7820edcbd94c39c0cd1aaf5cda286d24a3671cd40"}
Jul 16 22:14:40.523777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:40.523740    2557 generic.go:356] "Generic (PLEG): container finished" podID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerID="0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829" exitCode=0
Jul 16 22:14:40.523777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:40.523776    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" event={"ID":"c36a40b2-9a92-41cc-ba6b-15a84f6b520e","Type":"ContainerDied","Data":"0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829"}
Jul 16 22:14:41.531838 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:41.531778    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" event={"ID":"c36a40b2-9a92-41cc-ba6b-15a84f6b520e","Type":"ContainerStarted","Data":"f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088"}
Jul 16 22:14:41.559858 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:41.559788    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podStartSLOduration=2.649600278 podStartE2EDuration="5.559773105s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.910172827s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:41.556619683 +0000 UTC m=+1492.535968984" watchObservedRunningTime="2026-07-16 22:14:41.559773105 +0000 UTC m=+1492.539122406"
Jul 16 22:14:41.997476 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:41.997434    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:46.996603 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:46.996568    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:14:46.998045 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:46.998021    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:14:49.589827 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:49.589776    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:14:49.591297 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:49.591275    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:14:49.595273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:49.595252    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:14:49.596294 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:49.596277    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:14:56.997276 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:14:56.997233    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:15:00.678389 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.678351    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"]
Jul 16 22:15:00.682004 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.681980    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.684669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.684646    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 16 22:15:00.692781 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.692756    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"]
Jul 16 22:15:00.764798 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.764758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.765005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.764805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96fd0e-961c-4f3a-80f7-f9050d395091-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.765005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.764873    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.765005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.764893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.765005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.764967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866385 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lwt\" (UniqueName: \"kubernetes.io/projected/3a96fd0e-961c-4f3a-80f7-f9050d395091-kube-api-access-s9lwt\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866385 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866629 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866629 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866528    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96fd0e-961c-4f3a-80f7-f9050d395091-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866629 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866629 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866609    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.866945 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.867026 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.866951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.867065 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.867027    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.868833 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.868785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.869485 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.869462    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96fd0e-961c-4f3a-80f7-f9050d395091-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.967987 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.967884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lwt\" (UniqueName: \"kubernetes.io/projected/3a96fd0e-961c-4f3a-80f7-f9050d395091-kube-api-access-s9lwt\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:00.977020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:00.976990    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lwt\" (UniqueName: \"kubernetes.io/projected/3a96fd0e-961c-4f3a-80f7-f9050d395091-kube-api-access-s9lwt\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.108629 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.108594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.116067 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.116036    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.209709 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.209674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.209903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.209873    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.213026 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.212998    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.309207 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.309171    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a0f5e3d1545eedd31c6a3b629df7df5a5400a46392625f738ffdd5e141917306/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.325807 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.325780    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.493619 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.493580    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:05.639357 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.639327    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"]
Jul 16 22:15:05.640544 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:15:05.640515    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a96fd0e_961c_4f3a_80f7_f9050d395091.slice/crio-e8ae4f10d49cf2ea9bf266feb009faad1541c3b300fc290649030d34a4c8da22 WatchSource:0}: Error finding container e8ae4f10d49cf2ea9bf266feb009faad1541c3b300fc290649030d34a4c8da22: Status 404 returned error can't find the container with id e8ae4f10d49cf2ea9bf266feb009faad1541c3b300fc290649030d34a4c8da22
Jul 16 22:15:05.666865 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:05.666841    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" event={"ID":"3a96fd0e-961c-4f3a-80f7-f9050d395091","Type":"ContainerStarted","Data":"e8ae4f10d49cf2ea9bf266feb009faad1541c3b300fc290649030d34a4c8da22"}
Jul 16 22:15:06.675641 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:06.675602    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" event={"ID":"3a96fd0e-961c-4f3a-80f7-f9050d395091","Type":"ContainerStarted","Data":"de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97"}
Jul 16 22:15:06.701303 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:06.701236    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podStartSLOduration=6.701214377 podStartE2EDuration="6.701214377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:15:06.697175648 +0000 UTC m=+1517.676524959" watchObservedRunningTime="2026-07-16 22:15:06.701214377 +0000 UTC m=+1517.680563678"
Jul 16 22:15:06.997480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:06.997373    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:15:07.494710 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:07.494665    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:15.494126 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:15.494086    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:15:15.495681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:15.495650    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:15:16.997751 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:16.997702    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:15:25.495152 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:25.495097    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:15:26.997623 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:26.997569    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:15:35.494962 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:35.494909    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:15:36.997208 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:36.997145    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:15:45.494094 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:45.494049    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:15:46.997574 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:46.997526    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:15:55.494374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:55.494325    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:15:56.997134 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:15:56.997085    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:16:05.494195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:05.494140    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:16:06.997724 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:06.997679    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:16:15.494270 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:15.494221    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:16:16.997385 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:16.997338    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:16:25.494696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:25.494649    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:16:26.996896 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:26.996857    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:16:35.494941 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:35.494895    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:16:36.997336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:36.997286    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 16 22:16:45.494020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:45.493971    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:16:47.007564 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:47.007530    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:16:47.015632 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:47.015607    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:16:54.974102 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:54.974065    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"]
Jul 16 22:16:55.494398 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:16:55.494359    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:17:05.494227 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:05.494173    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:17:10.002698 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.002645    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main" containerID="cri-o://f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088" gracePeriod=45
Jul 16 22:17:10.960745 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.960720    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:17:10.976427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976400    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kserve-provision-location\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.976541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976436    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-dshm\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.976541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976487    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwq8h\" (UniqueName: \"kubernetes.io/projected/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kube-api-access-qwq8h\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.976541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976519    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-model-cache\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.976696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976559    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-home\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.976696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976618    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tls-certs\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.976696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.976671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tmp-dir\") pod \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\" (UID: \"c36a40b2-9a92-41cc-ba6b-15a84f6b520e\") "
Jul 16 22:17:10.977636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.977611    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:10.977830 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.977665    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-home" (OuterVolumeSpecName: "home") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:10.979213 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.979188    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-dshm" (OuterVolumeSpecName: "dshm") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:10.979396 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.979364    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:17:10.979396 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.979377    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kube-api-access-qwq8h" (OuterVolumeSpecName: "kube-api-access-qwq8h") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "kube-api-access-qwq8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:10.995562 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:10.995521    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:11.056709 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.056654    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c36a40b2-9a92-41cc-ba6b-15a84f6b520e" (UID: "c36a40b2-9a92-41cc-ba6b-15a84f6b520e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:11.077620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077592    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.077620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077621    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.077767 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077631    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.077767 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077639    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.077767 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077648    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwq8h\" (UniqueName: \"kubernetes.io/projected/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-kube-api-access-qwq8h\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.077767 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077656    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.077767 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.077664    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36a40b2-9a92-41cc-ba6b-15a84f6b520e-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.428748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.428716    2557 generic.go:356] "Generic (PLEG): container finished" podID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerID="f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088" exitCode=0
Jul 16 22:17:11.428934 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.428791    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"
Jul 16 22:17:11.428934 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.428790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" event={"ID":"c36a40b2-9a92-41cc-ba6b-15a84f6b520e","Type":"ContainerDied","Data":"f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088"}
Jul 16 22:17:11.428934 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.428921    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm" event={"ID":"c36a40b2-9a92-41cc-ba6b-15a84f6b520e","Type":"ContainerDied","Data":"94e56dd7b0ce252c756784b7820edcbd94c39c0cd1aaf5cda286d24a3671cd40"}
Jul 16 22:17:11.429057 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.428944    2557 scope.go:117] "RemoveContainer" containerID="f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088"
Jul 16 22:17:11.440315 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.440296    2557 scope.go:117] "RemoveContainer" containerID="0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829"
Jul 16 22:17:11.449234 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.449218    2557 scope.go:117] "RemoveContainer" containerID="f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088"
Jul 16 22:17:11.449462 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:17:11.449441    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088\": container with ID starting with f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088 not found: ID does not exist" containerID="f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088"
Jul 16 22:17:11.449544 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.449469    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088"} err="failed to get container status \"f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088\": rpc error: code = NotFound desc = could not find container \"f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088\": container with ID starting with f30b2da0075e94e670a1dcbd86aa05276c76f5894f9ac4afd6202da0db8d2088 not found: ID does not exist"
Jul 16 22:17:11.449544 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.449484    2557 scope.go:117] "RemoveContainer" containerID="0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829"
Jul 16 22:17:11.449734 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:17:11.449714    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829\": container with ID starting with 0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829 not found: ID does not exist" containerID="0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829"
Jul 16 22:17:11.449801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.449737    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829"} err="failed to get container status \"0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829\": rpc error: code = NotFound desc = could not find container \"0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829\": container with ID starting with 0c3e758dda25a111cb50bc92224b2dbe7b2f3ed67443008f17e534fa39b12829 not found: ID does not exist"
Jul 16 22:17:11.459983 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.459962    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"]
Jul 16 22:17:11.465470 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.465439    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bd7dd6dc4-zfhxm"]
Jul 16 22:17:11.562860 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:11.562805    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" path="/var/lib/kubelet/pods/c36a40b2-9a92-41cc-ba6b-15a84f6b520e/volumes"
Jul 16 22:17:15.504332 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:15.504300    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:17:15.512147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:15.512111    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:17:40.696758 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.696717    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"]
Jul 16 22:17:40.697220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.697149    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="storage-initializer"
Jul 16 22:17:40.697220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.697163    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="storage-initializer"
Jul 16 22:17:40.697220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.697189    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main"
Jul 16 22:17:40.697220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.697195    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main"
Jul 16 22:17:40.697354 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.697251    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c36a40b2-9a92-41cc-ba6b-15a84f6b520e" containerName="main"
Jul 16 22:17:40.701493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.701473    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.704248 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.704228    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:17:40.704373 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.704230    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 16 22:17:40.710863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.710838    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"]
Jul 16 22:17:40.816833 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.816785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5blvv\" (UniqueName: \"kubernetes.io/projected/70455c55-4f3e-44f9-ae93-de49d57b390c-kube-api-access-5blvv\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.816968 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.816853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-model-cache\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.816968 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.816889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-kserve-provision-location\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.816968 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.816959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-tmp-dir\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.817081 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.816981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-home\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.817081 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.817035    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-dshm\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.817081 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.817052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70455c55-4f3e-44f9-ae93-de49d57b390c-tls-certs\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917485 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blvv\" (UniqueName: \"kubernetes.io/projected/70455c55-4f3e-44f9-ae93-de49d57b390c-kube-api-access-5blvv\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-model-cache\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-kserve-provision-location\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917581    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-tmp-dir\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-home\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917786 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-dshm\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917786 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917665    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70455c55-4f3e-44f9-ae93-de49d57b390c-tls-certs\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-model-cache\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.917993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.917983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-tmp-dir\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.918110 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.918043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-kserve-provision-location\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.918110 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.918074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-home\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.920060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.920035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-dshm\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.920322 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.920303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70455c55-4f3e-44f9-ae93-de49d57b390c-tls-certs\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:40.926186 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:40.926166    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blvv\" (UniqueName: \"kubernetes.io/projected/70455c55-4f3e-44f9-ae93-de49d57b390c-kube-api-access-5blvv\") pod \"router-with-refs-test-kserve-7dc67694b-lh52n\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:41.012573 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:41.012518    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:41.158211 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:41.158184    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"]
Jul 16 22:17:41.158853 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:17:41.158827    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70455c55_4f3e_44f9_ae93_de49d57b390c.slice/crio-f3108aab3869eea9d59d65d4d47577aad6caa62f2165f2cc7fb35fa857d60c45 WatchSource:0}: Error finding container f3108aab3869eea9d59d65d4d47577aad6caa62f2165f2cc7fb35fa857d60c45: Status 404 returned error can't find the container with id f3108aab3869eea9d59d65d4d47577aad6caa62f2165f2cc7fb35fa857d60c45
Jul 16 22:17:41.584848 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:41.584800    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" event={"ID":"70455c55-4f3e-44f9-ae93-de49d57b390c","Type":"ContainerStarted","Data":"c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6"}
Jul 16 22:17:41.585039 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:41.584867    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" event={"ID":"70455c55-4f3e-44f9-ae93-de49d57b390c","Type":"ContainerStarted","Data":"f3108aab3869eea9d59d65d4d47577aad6caa62f2165f2cc7fb35fa857d60c45"}
Jul 16 22:17:43.325445 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:43.325403    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"]
Jul 16 22:17:44.602795 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:44.602712    2557 generic.go:356] "Generic (PLEG): container finished" podID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerID="c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6" exitCode=0
Jul 16 22:17:44.603167 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:44.602789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" event={"ID":"70455c55-4f3e-44f9-ae93-de49d57b390c","Type":"ContainerDied","Data":"c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6"}
Jul 16 22:17:45.609861 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:45.609801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" event={"ID":"70455c55-4f3e-44f9-ae93-de49d57b390c","Type":"ContainerStarted","Data":"de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98"}
Jul 16 22:17:45.631530 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:45.631466    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podStartSLOduration=2.656848282 podStartE2EDuration="5.631443171s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.974594889s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:45.629136307 +0000 UTC m=+1676.608485606" watchObservedRunningTime="2026-07-16 22:17:45.631443171 +0000 UTC m=+1676.610792477"
Jul 16 22:17:46.013306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:46.013273    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:51.013496 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:51.013454    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:17:51.014683 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:51.014657    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:17:58.352907 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:58.352832    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" containerID="cri-o://de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97" gracePeriod=45
Jul 16 22:17:58.505868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:58.505828    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 16 22:17:59.112865 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.112835    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:17:59.287520 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287487    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-home\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.287699 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287529    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-tmp-dir\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.287699 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287637    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.287699 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287670    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96fd0e-961c-4f3a-80f7-f9050d395091-tls-certs\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.287909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287707    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-dshm\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.287909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287786    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-model-cache\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.287909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.287846    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9lwt\" (UniqueName: \"kubernetes.io/projected/3a96fd0e-961c-4f3a-80f7-f9050d395091-kube-api-access-s9lwt\") pod \"3a96fd0e-961c-4f3a-80f7-f9050d395091\" (UID: \"3a96fd0e-961c-4f3a-80f7-f9050d395091\") "
Jul 16 22:17:59.288497 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.288124    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-home" (OuterVolumeSpecName: "home") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:59.288497 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.288335    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:59.290265 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.290227    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a96fd0e-961c-4f3a-80f7-f9050d395091-kube-api-access-s9lwt" (OuterVolumeSpecName: "kube-api-access-s9lwt") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "kube-api-access-s9lwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:59.290587 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.290565    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-dshm" (OuterVolumeSpecName: "dshm") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:59.290661 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.290572    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a96fd0e-961c-4f3a-80f7-f9050d395091-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:17:59.292230 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.292211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8" (OuterVolumeSpecName: "kserve-pvc-source") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 22:17:59.305495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.305465    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a96fd0e-961c-4f3a-80f7-f9050d395091" (UID: "3a96fd0e-961c-4f3a-80f7-f9050d395091"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:59.388834 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388788    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.388834 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388825    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.389245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388854    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-136-80.ec2.internal\" "
Jul 16 22:17:59.389245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388870    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96fd0e-961c-4f3a-80f7-f9050d395091-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.389245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388884    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.389245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388896    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96fd0e-961c-4f3a-80f7-f9050d395091-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.389245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.388904    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lwt\" (UniqueName: \"kubernetes.io/projected/3a96fd0e-961c-4f3a-80f7-f9050d395091-kube-api-access-s9lwt\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.415366 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.415348    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8") on node "ip-10-0-136-80.ec2.internal"
Jul 16 22:17:59.489392 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.489362    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\""
Jul 16 22:17:59.696966 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.696932    2557 generic.go:356] "Generic (PLEG): container finished" podID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerID="de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97" exitCode=0
Jul 16 22:17:59.697166 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.697009    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"
Jul 16 22:17:59.697166 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.697026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" event={"ID":"3a96fd0e-961c-4f3a-80f7-f9050d395091","Type":"ContainerDied","Data":"de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97"}
Jul 16 22:17:59.697166 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.697092    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww" event={"ID":"3a96fd0e-961c-4f3a-80f7-f9050d395091","Type":"ContainerDied","Data":"e8ae4f10d49cf2ea9bf266feb009faad1541c3b300fc290649030d34a4c8da22"}
Jul 16 22:17:59.697166 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.697115    2557 scope.go:117] "RemoveContainer" containerID="de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97"
Jul 16 22:17:59.706619 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.706599    2557 scope.go:117] "RemoveContainer" containerID="de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97"
Jul 16 22:17:59.706886 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:17:59.706866    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97\": container with ID starting with de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97 not found: ID does not exist" containerID="de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97"
Jul 16 22:17:59.706947 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.706894    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97"} err="failed to get container status \"de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97\": rpc error: code = NotFound desc = could not find container \"de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97\": container with ID starting with de83dc0bc0775f310ed7c7852cb7cf4da35c4db9f434aca4b2d7b9e592814f97 not found: ID does not exist"
Jul 16 22:17:59.715330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.715305    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"]
Jul 16 22:17:59.718637 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:17:59.718618    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dxd8ww"]
Jul 16 22:18:01.012998 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:01.012960    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:18:01.562345 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:01.562318    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" path="/var/lib/kubelet/pods/3a96fd0e-961c-4f3a-80f7-f9050d395091/volumes"
Jul 16 22:18:08.978536 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.978507    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"]
Jul 16 22:18:08.978912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.978890    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main"
Jul 16 22:18:08.978912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.978901    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main"
Jul 16 22:18:08.978998 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.978964    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a96fd0e-961c-4f3a-80f7-f9050d395091" containerName="main"
Jul 16 22:18:08.982344 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.982327    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:08.989094 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.989075    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 16 22:18:08.989201 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.989166    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-kwt89\""
Jul 16 22:18:09.000011 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:08.999991    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"]
Jul 16 22:18:09.002484 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.002466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.004630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.004608    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"]
Jul 16 22:18:09.040459 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.040438    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"]
Jul 16 22:18:09.175799 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.175765    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.175799 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.175807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.176059 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.175905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.176059 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.175952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.176059 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176002    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m624\" (UniqueName: \"kubernetes.io/projected/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-kube-api-access-8m624\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.176059 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.176222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.176222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.176222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176122    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.176222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bmw\" (UniqueName: \"kubernetes.io/projected/6edd9a28-528f-48f9-9f0f-bc26651a5d92-kube-api-access-q5bmw\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.176222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.176406 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.176242    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.277170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277154    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m624\" (UniqueName: \"kubernetes.io/projected/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-kube-api-access-8m624\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bmw\" (UniqueName: \"kubernetes.io/projected/6edd9a28-528f-48f9-9f0f-bc26651a5d92-kube-api-access-q5bmw\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277493 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277910 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.277910 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277552    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.277910 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277749    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.277910 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.278151 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.277892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.278151 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.278120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.278151 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.278130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.278479 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.278225    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.280287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.280263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.280449 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.280432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.280870 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.280849    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:09.281153 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.281137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.287438 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.287413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m624\" (UniqueName: \"kubernetes.io/projected/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-kube-api-access-8m624\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:09.288496 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:09.288471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bmw\" (UniqueName: \"kubernetes.io/projected/6edd9a28-528f-48f9-9f0f-bc26651a5d92-kube-api-access-q5bmw\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:11.013098 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:11.013002    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:18:17.658435 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.658396    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.666444 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.666421    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.759460 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.759421    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.759613 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.759549    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.762512 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.762487    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.848308 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.848278    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a0f5e3d1545eedd31c6a3b629df7df5a5400a46392625f738ffdd5e141917306/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.864802 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.864771    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:17.961231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.961128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:17.961416 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.961275    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:17.964251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.964228    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:17.976472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.976448    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:17.991881 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:17.991860    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:18.012524 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.012496    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:18.151610 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.151578    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"]
Jul 16 22:18:18.153550 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:18:18.153524    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edd9a28_528f_48f9_9f0f_bc26651a5d92.slice/crio-5a9e4675dd1ff43ebdd2841bc48c53551c8356a0541ee9e13d62b77b88ffa9e2 WatchSource:0}: Error finding container 5a9e4675dd1ff43ebdd2841bc48c53551c8356a0541ee9e13d62b77b88ffa9e2: Status 404 returned error can't find the container with id 5a9e4675dd1ff43ebdd2841bc48c53551c8356a0541ee9e13d62b77b88ffa9e2
Jul 16 22:18:18.155945 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.155851    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:18:18.170944 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.170924    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"]
Jul 16 22:18:18.172215 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:18:18.172192    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1508a08f_4dcf_4dbc_97cf_97b7bce8aa16.slice/crio-8dc36ffe0340eeca9c2ef2fdd71cdb9d2b9af8c6f43ad23832abc3b80ab8916b WatchSource:0}: Error finding container 8dc36ffe0340eeca9c2ef2fdd71cdb9d2b9af8c6f43ad23832abc3b80ab8916b: Status 404 returned error can't find the container with id 8dc36ffe0340eeca9c2ef2fdd71cdb9d2b9af8c6f43ad23832abc3b80ab8916b
Jul 16 22:18:18.802799 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.802763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" event={"ID":"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16","Type":"ContainerStarted","Data":"eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c"}
Jul 16 22:18:18.803288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.802849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" event={"ID":"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16","Type":"ContainerStarted","Data":"8dc36ffe0340eeca9c2ef2fdd71cdb9d2b9af8c6f43ad23832abc3b80ab8916b"}
Jul 16 22:18:18.804076 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.804055    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" event={"ID":"6edd9a28-528f-48f9-9f0f-bc26651a5d92","Type":"ContainerStarted","Data":"5a9e4675dd1ff43ebdd2841bc48c53551c8356a0541ee9e13d62b77b88ffa9e2"}
Jul 16 22:18:18.830339 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:18.829892    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podStartSLOduration=10.829875098 podStartE2EDuration="10.829875098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:18.825844207 +0000 UTC m=+1709.805193498" watchObservedRunningTime="2026-07-16 22:18:18.829875098 +0000 UTC m=+1709.809224395"
Jul 16 22:18:20.817526 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:20.817487    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" event={"ID":"6edd9a28-528f-48f9-9f0f-bc26651a5d92","Type":"ContainerStarted","Data":"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"}
Jul 16 22:18:20.817959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:20.817549    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:21.013753 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:21.013716    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:18:21.827685 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:21.827644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" event={"ID":"6edd9a28-528f-48f9-9f0f-bc26651a5d92","Type":"ContainerStarted","Data":"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"}
Jul 16 22:18:21.874351 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:21.874288    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podStartSLOduration=12.120503029 podStartE2EDuration="13.874255898s" totalImagesPullingTime="1.753752869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:21.873803147 +0000 UTC m=+1712.853152449" watchObservedRunningTime="2026-07-16 22:18:21.874255898 +0000 UTC m=+1712.853605198"
Jul 16 22:18:27.992333 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:27.992295    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:27.992946 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:27.992365    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:27.994215 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:27.994188    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:18:28.013293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:28.013266    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:28.013426 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:28.013341    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:18:28.015036 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:28.015007    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:18:31.013285 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:31.013229    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:18:37.992599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:37.992540    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:18:38.012873 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:38.012832    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:18:38.023319 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:38.023294    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:18:41.013842 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:41.013787    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:18:47.993021 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:47.992580    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:18:48.013673 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:48.013626    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:18:51.014030 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:51.013982    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:18:57.992857 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:57.992787    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:18:58.013579 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:18:58.013518    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:19:01.013512 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:01.013455    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:19:07.992733 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:07.992679    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:19:08.013938 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:08.013892    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:19:11.013713 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:11.013662    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:19:17.992631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:17.992582    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:19:18.012803 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:18.012755    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:19:21.013262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:21.013210    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:19:27.992555 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:27.992491    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:19:28.013386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:28.013346    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:19:31.013632 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:31.013591    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:19:37.992970 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:37.992914    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:19:38.013559 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:38.013523    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:19:41.023278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:41.023199    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:19:41.030863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:41.030839    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:19:47.992644 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:47.992600    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:19:48.013459 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:48.013426    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:19:49.632286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:49.632254    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:19:49.634553 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:49.634524    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:19:49.637615 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:49.637594    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:19:49.639474 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:49.639453    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:19:51.681246 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:51.681208    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"]
Jul 16 22:19:57.993053 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:57.992931    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:19:58.013682 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:19:58.013642    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:20:06.705789 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:06.705738    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" containerID="cri-o://de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98" gracePeriod=45
Jul 16 22:20:07.024914 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:07.024788    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 16 22:20:07.886894 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:07.886867    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:20:07.992663 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:07.992567    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:20:08.008612 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008574    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-kserve-provision-location\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.008612 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008620    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5blvv\" (UniqueName: \"kubernetes.io/projected/70455c55-4f3e-44f9-ae93-de49d57b390c-kube-api-access-5blvv\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.008930 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008660    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-home\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.008930 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008686    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-model-cache\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.008930 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008751    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70455c55-4f3e-44f9-ae93-de49d57b390c-tls-certs\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.008930 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008772    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-tmp-dir\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.008930 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.008796    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-dshm\") pod \"70455c55-4f3e-44f9-ae93-de49d57b390c\" (UID: \"70455c55-4f3e-44f9-ae93-de49d57b390c\") "
Jul 16 22:20:08.009214 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.009119    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-model-cache" (OuterVolumeSpecName: "model-cache") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:08.009448 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.009421    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-home" (OuterVolumeSpecName: "home") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:08.011716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.011683    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-dshm" (OuterVolumeSpecName: "dshm") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:08.012381 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.012351    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70455c55-4f3e-44f9-ae93-de49d57b390c-kube-api-access-5blvv" (OuterVolumeSpecName: "kube-api-access-5blvv") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "kube-api-access-5blvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:20:08.012574 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.012554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70455c55-4f3e-44f9-ae93-de49d57b390c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:20:08.012899 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.012874    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:20:08.022098 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.022072    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:08.076795 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.076745    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70455c55-4f3e-44f9-ae93-de49d57b390c" (UID: "70455c55-4f3e-44f9-ae93-de49d57b390c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:08.110226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110181    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.110226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110223    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70455c55-4f3e-44f9-ae93-de49d57b390c-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.110226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110235    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.110421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110247    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.110421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110258    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.110421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110270    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5blvv\" (UniqueName: \"kubernetes.io/projected/70455c55-4f3e-44f9-ae93-de49d57b390c-kube-api-access-5blvv\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.110421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.110281    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70455c55-4f3e-44f9-ae93-de49d57b390c-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:20:08.536999 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.536960    2557 generic.go:356] "Generic (PLEG): container finished" podID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerID="de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98" exitCode=0
Jul 16 22:20:08.537177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.537039    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"
Jul 16 22:20:08.537177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.537035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" event={"ID":"70455c55-4f3e-44f9-ae93-de49d57b390c","Type":"ContainerDied","Data":"de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98"}
Jul 16 22:20:08.537177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.537124    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n" event={"ID":"70455c55-4f3e-44f9-ae93-de49d57b390c","Type":"ContainerDied","Data":"f3108aab3869eea9d59d65d4d47577aad6caa62f2165f2cc7fb35fa857d60c45"}
Jul 16 22:20:08.537177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.537151    2557 scope.go:117] "RemoveContainer" containerID="de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98"
Jul 16 22:20:08.548754 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.548738    2557 scope.go:117] "RemoveContainer" containerID="c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6"
Jul 16 22:20:08.557726 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.557706    2557 scope.go:117] "RemoveContainer" containerID="de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98"
Jul 16 22:20:08.558078 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:20:08.558059    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98\": container with ID starting with de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98 not found: ID does not exist" containerID="de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98"
Jul 16 22:20:08.558152 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.558087    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98"} err="failed to get container status \"de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98\": rpc error: code = NotFound desc = could not find container \"de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98\": container with ID starting with de2652039a25f063a843d4d7504b640e6223e249dd258562e680b3ff6537fb98 not found: ID does not exist"
Jul 16 22:20:08.558152 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.558105    2557 scope.go:117] "RemoveContainer" containerID="c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6"
Jul 16 22:20:08.558366 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:20:08.558343    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6\": container with ID starting with c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6 not found: ID does not exist" containerID="c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6"
Jul 16 22:20:08.558463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.558368    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6"} err="failed to get container status \"c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6\": rpc error: code = NotFound desc = could not find container \"c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6\": container with ID starting with c0dee620e1870f9adc86a3a747ef4d7e00e1da1c321c285126b12f0bd9b24dc6 not found: ID does not exist"
Jul 16 22:20:08.562970 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.562949    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"]
Jul 16 22:20:08.568213 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:08.568192    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dc67694b-lh52n"]
Jul 16 22:20:09.562338 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:09.562305    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" path="/var/lib/kubelet/pods/70455c55-4f3e-44f9-ae93-de49d57b390c/volumes"
Jul 16 22:20:17.992464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:17.992410    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:20:18.013851 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:18.013793    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:20:20.287736 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.287702    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"]
Jul 16 22:20:20.288178 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.288161    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main"
Jul 16 22:20:20.288231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.288182    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main"
Jul 16 22:20:20.288231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.288208    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="storage-initializer"
Jul 16 22:20:20.288231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.288213    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="storage-initializer"
Jul 16 22:20:20.288327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.288279    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="70455c55-4f3e-44f9-ae93-de49d57b390c" containerName="main"
Jul 16 22:20:20.291457 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.291441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.295457 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.295429    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:20:20.295935 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.295909    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-w9ntm\""
Jul 16 22:20:20.296951 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.296931    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 16 22:20:20.299330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.299060    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"]
Jul 16 22:20:20.302709 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.302692    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.320382 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.320363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"]
Jul 16 22:20:20.324332 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.324312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"]
Jul 16 22:20:20.426447 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426420    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.426600 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.426600 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r94\" (UniqueName: \"kubernetes.io/projected/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kube-api-access-59r94\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.426600 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426505    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.426600 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.426600 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.426779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426604    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.426779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426650    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.426779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426677    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.426779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426696    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.426779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.426779 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426734    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.427000 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bm89\" (UniqueName: \"kubernetes.io/projected/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kube-api-access-4bm89\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.427000 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.426837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527453 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527426    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bm89\" (UniqueName: \"kubernetes.io/projected/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kube-api-access-4bm89\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527662    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r94\" (UniqueName: \"kubernetes.io/projected/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kube-api-access-59r94\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.527959 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.527832    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.528323 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.528323 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.528323 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528245    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.528495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528416    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.528495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528451    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.528495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.528495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.528884 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.528497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.530623 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.530597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.530760 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.530743    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.530853 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.530838    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.531366 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.531345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.547117 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.547056    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r94\" (UniqueName: \"kubernetes.io/projected/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kube-api-access-59r94\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.547755 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.547733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bm89\" (UniqueName: \"kubernetes.io/projected/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kube-api-access-4bm89\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.601639 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.601615    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:20.612091 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.612069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:20.795673 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.795630    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"]
Jul 16 22:20:20.796499 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:20:20.796470    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fa2ad0_8e98_40f5_9af2_09f2cd5c7c91.slice/crio-e9b7b1c7c1bf211679a511b6bbf41f77a69f3d0a8be763d1cdaee60bb0cf4bc0 WatchSource:0}: Error finding container e9b7b1c7c1bf211679a511b6bbf41f77a69f3d0a8be763d1cdaee60bb0cf4bc0: Status 404 returned error can't find the container with id e9b7b1c7c1bf211679a511b6bbf41f77a69f3d0a8be763d1cdaee60bb0cf4bc0
Jul 16 22:20:20.809746 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:20.809724    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"]
Jul 16 22:20:20.810879 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:20:20.810857    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb386b_a783_41bc_8da7_8ef16018f6a3.slice/crio-90494f01200221e98bdf7c8b5720d4d11f68a19318ea386421565ced2b0afabf WatchSource:0}: Error finding container 90494f01200221e98bdf7c8b5720d4d11f68a19318ea386421565ced2b0afabf: Status 404 returned error can't find the container with id 90494f01200221e98bdf7c8b5720d4d11f68a19318ea386421565ced2b0afabf
Jul 16 22:20:21.611147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:21.611107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" event={"ID":"f7bb386b-a783-41bc-8da7-8ef16018f6a3","Type":"ContainerStarted","Data":"cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52"}
Jul 16 22:20:21.611610 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:21.611168    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" event={"ID":"f7bb386b-a783-41bc-8da7-8ef16018f6a3","Type":"ContainerStarted","Data":"90494f01200221e98bdf7c8b5720d4d11f68a19318ea386421565ced2b0afabf"}
Jul 16 22:20:21.612685 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:21.612658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerStarted","Data":"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"}
Jul 16 22:20:21.612795 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:21.612701    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerStarted","Data":"e9b7b1c7c1bf211679a511b6bbf41f77a69f3d0a8be763d1cdaee60bb0cf4bc0"}
Jul 16 22:20:21.612862 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:21.612800    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:22.620176 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:22.620141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerStarted","Data":"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"}
Jul 16 22:20:24.652096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:24.652058    2557 generic.go:356] "Generic (PLEG): container finished" podID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerID="cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52" exitCode=0
Jul 16 22:20:24.652681 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:24.652350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" event={"ID":"f7bb386b-a783-41bc-8da7-8ef16018f6a3","Type":"ContainerDied","Data":"cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52"}
Jul 16 22:20:25.660497 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:25.660456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" event={"ID":"f7bb386b-a783-41bc-8da7-8ef16018f6a3","Type":"ContainerStarted","Data":"fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d"}
Jul 16 22:20:25.662426 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:25.662395    2557 generic.go:356] "Generic (PLEG): container finished" podID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerID="4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a" exitCode=0
Jul 16 22:20:25.662552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:25.662459    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerDied","Data":"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"}
Jul 16 22:20:25.690864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:25.690800    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podStartSLOduration=2.500065654 podStartE2EDuration="5.690783962s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.190718308s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:25.688235407 +0000 UTC m=+1836.667584722" watchObservedRunningTime="2026-07-16 22:20:25.690783962 +0000 UTC m=+1836.670133260"
Jul 16 22:20:26.673608 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:26.673559    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerStarted","Data":"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"}
Jul 16 22:20:26.704262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:26.704206    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podStartSLOduration=3.452107173 podStartE2EDuration="6.704191345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.252084172s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:26.700286157 +0000 UTC m=+1837.679635463" watchObservedRunningTime="2026-07-16 22:20:26.704191345 +0000 UTC m=+1837.683540643"
Jul 16 22:20:27.992758 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:27.992705    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:20:28.013198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:28.013150    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:20:30.602599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.602554    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:30.603115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.602645    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:30.604427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.604384    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:20:30.612856 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.612826    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:30.612993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.612888    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:20:30.614307 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.614282    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:20:30.621912 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:30.621891    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:20:37.993009 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:37.992950    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:20:38.013982 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:38.013935    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:20:40.602239 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:40.602192    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:20:40.614369 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:40.614322    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:20:47.992966 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:47.992846    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:20:48.013642 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:48.013597    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:20:50.602801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:50.602745    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:20:50.612988 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:50.612936    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:20:57.993306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:57.993240    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Jul 16 22:20:58.013069 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:20:58.013024    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 16 22:21:00.602303 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:00.602244    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:21:00.612981 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:00.612922    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:21:08.002846 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:08.002794    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:21:08.021921 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:08.021890    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:21:08.024543 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:08.024508    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:21:08.032615 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:08.032585    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:21:10.602748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:10.602635    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:21:10.612729 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:10.612676    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:21:16.723518 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:16.723477    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"]
Jul 16 22:21:16.728372 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:16.728343    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"]
Jul 16 22:21:20.602579 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:20.602526    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:21:20.613988 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:20.613944    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:21:30.602836 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:30.602771    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:21:30.614163 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:30.614117    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:21:31.759884 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:31.759779    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main" containerID="cri-o://c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06" gracePeriod=45
Jul 16 22:21:31.764293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:31.764252    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main" containerID="cri-o://eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c" gracePeriod=45
Jul 16 22:21:32.584156 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.584117    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="llm-d-routing-sidecar" containerID="cri-o://813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106" gracePeriod=45
Jul 16 22:21:32.773620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.773594    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:21:32.826095 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826007    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-model-cache\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.826095 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826050    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m624\" (UniqueName: \"kubernetes.io/projected/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-kube-api-access-8m624\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.826339 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tmp-dir\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.826339 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826280    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-model-cache" (OuterVolumeSpecName: "model-cache") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.826449 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826398    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-home\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.826449 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826441    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-dshm\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.826595 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826574    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.826786 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.826769    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tls-certs\") pod \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\" (UID: \"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16\") "
Jul 16 22:21:32.827040 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.827011    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-home" (OuterVolumeSpecName: "home") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.827362 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.827344    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.827470 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.827459    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.829889 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.829764    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-dshm" (OuterVolumeSpecName: "dshm") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.830318 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.830287    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:21:32.830609 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.830581    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-kube-api-access-8m624" (OuterVolumeSpecName: "kube-api-access-8m624") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "kube-api-access-8m624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:21:32.833225 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.833199    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8" (OuterVolumeSpecName: "kserve-pvc-source") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 22:21:32.840020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.839974    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" (UID: "1508a08f-4dcf-4dbc-97cf-97b7bce8aa16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.852272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.852243    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:21:32.928247 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928211    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-model-cache\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928247 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928251    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tmp-dir\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5bmw\" (UniqueName: \"kubernetes.io/projected/6edd9a28-528f-48f9-9f0f-bc26651a5d92-kube-api-access-q5bmw\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928309    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tls-certs\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928397    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-home\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928474    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928507    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-dshm\") pod \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\" (UID: \"6edd9a28-528f-48f9-9f0f-bc26651a5d92\") "
Jul 16 22:21:32.928769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928505    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-model-cache" (OuterVolumeSpecName: "model-cache") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.928769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928711    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.928769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928729    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.928769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928742    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.928769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928756    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.928769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.928768    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m624\" (UniqueName: \"kubernetes.io/projected/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16-kube-api-access-8m624\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.929106 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.929015    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-home" (OuterVolumeSpecName: "home") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.932055 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.932017    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edd9a28-528f-48f9-9f0f-bc26651a5d92-kube-api-access-q5bmw" (OuterVolumeSpecName: "kube-api-access-q5bmw") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "kube-api-access-q5bmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:21:32.932393 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.932364    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8" (OuterVolumeSpecName: "kserve-pvc-source") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 22:21:32.932721 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.932697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:21:32.936358 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.936324    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-dshm" (OuterVolumeSpecName: "dshm") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.941751 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:32.941722    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6edd9a28-528f-48f9-9f0f-bc26651a5d92" (UID: "6edd9a28-528f-48f9-9f0f-bc26651a5d92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:33.029851 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.029795    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bmw\" (UniqueName: \"kubernetes.io/projected/6edd9a28-528f-48f9-9f0f-bc26651a5d92-kube-api-access-q5bmw\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:33.029851 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.029843    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:33.029851 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.029854    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:33.030096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.029887    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-136-80.ec2.internal\" "
Jul 16 22:21:33.030096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.029897    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:33.030096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.029905    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd9a28-528f-48f9-9f0f-bc26651a5d92-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:21:33.068993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.068966    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8") on node "ip-10-0-136-80.ec2.internal"
Jul 16 22:21:33.130431 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.130341    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\""
Jul 16 22:21:33.146088 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.146055    2557 generic.go:356] "Generic (PLEG): container finished" podID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerID="eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c" exitCode=0
Jul 16 22:21:33.146288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.146118    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"
Jul 16 22:21:33.146288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.146138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" event={"ID":"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16","Type":"ContainerDied","Data":"eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c"}
Jul 16 22:21:33.146288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.146234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws" event={"ID":"1508a08f-4dcf-4dbc-97cf-97b7bce8aa16","Type":"ContainerDied","Data":"8dc36ffe0340eeca9c2ef2fdd71cdb9d2b9af8c6f43ad23832abc3b80ab8916b"}
Jul 16 22:21:33.146288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.146271    2557 scope.go:117] "RemoveContainer" containerID="eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c"
Jul 16 22:21:33.148357 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.148329    2557 generic.go:356] "Generic (PLEG): container finished" podID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerID="c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06" exitCode=0
Jul 16 22:21:33.148357 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.148355    2557 generic.go:356] "Generic (PLEG): container finished" podID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerID="813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106" exitCode=0
Jul 16 22:21:33.148560 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.148421    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"
Jul 16 22:21:33.148626 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.148413    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" event={"ID":"6edd9a28-528f-48f9-9f0f-bc26651a5d92","Type":"ContainerDied","Data":"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"}
Jul 16 22:21:33.148687 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.148646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" event={"ID":"6edd9a28-528f-48f9-9f0f-bc26651a5d92","Type":"ContainerDied","Data":"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"}
Jul 16 22:21:33.148687 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.148682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g" event={"ID":"6edd9a28-528f-48f9-9f0f-bc26651a5d92","Type":"ContainerDied","Data":"5a9e4675dd1ff43ebdd2841bc48c53551c8356a0541ee9e13d62b77b88ffa9e2"}
Jul 16 22:21:33.158293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.158264    2557 scope.go:117] "RemoveContainer" containerID="eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c"
Jul 16 22:21:33.158690 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:21:33.158666    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c\": container with ID starting with eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c not found: ID does not exist" containerID="eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c"
Jul 16 22:21:33.158795 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.158703    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c"} err="failed to get container status \"eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c\": rpc error: code = NotFound desc = could not find container \"eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c\": container with ID starting with eef1d09c411b62e807de9e0c4b16ca7b50e13f9a471ba280db0bac48c672774c not found: ID does not exist"
Jul 16 22:21:33.158795 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.158728    2557 scope.go:117] "RemoveContainer" containerID="c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"
Jul 16 22:21:33.169412 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.169390    2557 scope.go:117] "RemoveContainer" containerID="813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"
Jul 16 22:21:33.177552 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.177521    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"]
Jul 16 22:21:33.181877 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.181857    2557 scope.go:117] "RemoveContainer" containerID="c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"
Jul 16 22:21:33.182237 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:21:33.182217    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06\": container with ID starting with c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06 not found: ID does not exist" containerID="c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"
Jul 16 22:21:33.182327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182249    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"} err="failed to get container status \"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06\": rpc error: code = NotFound desc = could not find container \"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06\": container with ID starting with c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06 not found: ID does not exist"
Jul 16 22:21:33.182327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182272    2557 scope.go:117] "RemoveContainer" containerID="813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"
Jul 16 22:21:33.182327 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182301    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-vs44g"]
Jul 16 22:21:33.182587 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:21:33.182568    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106\": container with ID starting with 813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106 not found: ID does not exist" containerID="813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"
Jul 16 22:21:33.182660 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182595    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"} err="failed to get container status \"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106\": rpc error: code = NotFound desc = could not find container \"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106\": container with ID starting with 813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106 not found: ID does not exist"
Jul 16 22:21:33.182660 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182615    2557 scope.go:117] "RemoveContainer" containerID="c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"
Jul 16 22:21:33.182964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182933    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06"} err="failed to get container status \"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06\": rpc error: code = NotFound desc = could not find container \"c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06\": container with ID starting with c3550e0db436ab707124a6893f2e9c41265ecd1d285e9de5ee24808ab3558a06 not found: ID does not exist"
Jul 16 22:21:33.183036 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.182968    2557 scope.go:117] "RemoveContainer" containerID="813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"
Jul 16 22:21:33.183281 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.183257    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106"} err="failed to get container status \"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106\": rpc error: code = NotFound desc = could not find container \"813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106\": container with ID starting with 813600e710f3f30346e4f2c31900a2647eacdec2a49d8a5b45c91bade074a106 not found: ID does not exist"
Jul 16 22:21:33.195628 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.195604    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"]
Jul 16 22:21:33.202870 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.202844    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-797zsws"]
Jul 16 22:21:33.564005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.563961    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" path="/var/lib/kubelet/pods/1508a08f-4dcf-4dbc-97cf-97b7bce8aa16/volumes"
Jul 16 22:21:33.564608 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:33.564579    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" path="/var/lib/kubelet/pods/6edd9a28-528f-48f9-9f0f-bc26651a5d92/volumes"
Jul 16 22:21:40.602451 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:40.602407    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:21:40.612633 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:40.612584    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:21:49.227145 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227105    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227492    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main"
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227502    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main"
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227512    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main"
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227517    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main"
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227537    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="llm-d-routing-sidecar"
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227543    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="llm-d-routing-sidecar"
Jul 16 22:21:49.227601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227601    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="llm-d-routing-sidecar"
Jul 16 22:21:49.227850 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227610    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1508a08f-4dcf-4dbc-97cf-97b7bce8aa16" containerName="main"
Jul 16 22:21:49.227850 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.227619    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edd9a28-528f-48f9-9f0f-bc26651a5d92" containerName="main"
Jul 16 22:21:49.229772 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.229757    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.235242 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.235219    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 16 22:21:49.236594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.236570    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-gc6pd\""
Jul 16 22:21:49.251767 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.251739    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 22:21:49.303222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.303187    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 22:21:49.306206 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.306186    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.322267 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.322237    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 22:21:49.382735 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.382693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqtvd\" (UniqueName: \"kubernetes.io/projected/783c323c-d2d9-46d8-88ea-3c7afe0a238c-kube-api-access-pqtvd\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.382950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.382748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.382950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.382832    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.382950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.382858    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.382950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.382882    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.382950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.382929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.483843 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.483728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqtvd\" (UniqueName: \"kubernetes.io/projected/783c323c-d2d9-46d8-88ea-3c7afe0a238c-kube-api-access-pqtvd\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.483843 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.483796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.484042 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.483856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484042 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.483896    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.484042 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484026    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484141 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484071    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484141 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.484220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484315 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.484315 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484315 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484274    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801e1d03-9641-4637-a49b-41b35f8e0b9c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.484453 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484348    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdtg\" (UniqueName: \"kubernetes.io/projected/801e1d03-9641-4637-a49b-41b35f8e0b9c-kube-api-access-tjdtg\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.484542 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484524    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.484620 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.484598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.486345 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.486307    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.486464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.486418    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 16 22:21:49.493359 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.493328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqtvd\" (UniqueName: \"kubernetes.io/projected/783c323c-d2d9-46d8-88ea-3c7afe0a238c-kube-api-access-pqtvd\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.497403 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.497374    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:49.585556 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.585525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdtg\" (UniqueName: \"kubernetes.io/projected/801e1d03-9641-4637-a49b-41b35f8e0b9c-kube-api-access-tjdtg\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.585748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.585580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.585748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.585619    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.585748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.585667    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.585748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.585703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.585748 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.585728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801e1d03-9641-4637-a49b-41b35f8e0b9c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.586021 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.586000    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.586119 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.586100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.586174 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.586125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.588095 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.588068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.588197 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.588130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801e1d03-9641-4637-a49b-41b35f8e0b9c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:49.594622 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:49.594599    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdtg\" (UniqueName: \"kubernetes.io/projected/801e1d03-9641-4637-a49b-41b35f8e0b9c-kube-api-access-tjdtg\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:50.602831 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:50.602768    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:21:50.614121 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:50.614078    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:21:51.809085 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:51.809045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:51.815137 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:51.815106    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:51.909586 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:51.909552    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:51.909801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:51.909711    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:51.912689 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:51.912658    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:52.002108 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.002070    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a0f5e3d1545eedd31c6a3b629df7df5a5400a46392625f738ffdd5e141917306/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:52.014703 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.014671    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:52.111421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.111308    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:52.111611 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.111434    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:52.114566 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.114539    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:52.126920 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.126888    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:52.250330 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.243542    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-gc6pd\""
Jul 16 22:21:52.250680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.250648    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:21:52.316284 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.316225    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:21:52.409946 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.409902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 22:21:52.413341 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:21:52.413291    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783c323c_d2d9_46d8_88ea_3c7afe0a238c.slice/crio-44b395bf9083e2431f66267ba2c4528bb748ab4967cda2a28248e444ea7e82f4 WatchSource:0}: Error finding container 44b395bf9083e2431f66267ba2c4528bb748ab4967cda2a28248e444ea7e82f4: Status 404 returned error can't find the container with id 44b395bf9083e2431f66267ba2c4528bb748ab4967cda2a28248e444ea7e82f4
Jul 16 22:21:52.475891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:52.475867    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 22:21:52.478651 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:21:52.478627    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801e1d03_9641_4637_a49b_41b35f8e0b9c.slice/crio-54db1a05329b1e74f7235f3f717bc01c826f7e84a5f1eea3b289d82bd94e04f7 WatchSource:0}: Error finding container 54db1a05329b1e74f7235f3f717bc01c826f7e84a5f1eea3b289d82bd94e04f7: Status 404 returned error can't find the container with id 54db1a05329b1e74f7235f3f717bc01c826f7e84a5f1eea3b289d82bd94e04f7
Jul 16 22:21:53.288886 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:53.288849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"783c323c-d2d9-46d8-88ea-3c7afe0a238c","Type":"ContainerStarted","Data":"57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2"}
Jul 16 22:21:53.289419 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:53.288913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"783c323c-d2d9-46d8-88ea-3c7afe0a238c","Type":"ContainerStarted","Data":"44b395bf9083e2431f66267ba2c4528bb748ab4967cda2a28248e444ea7e82f4"}
Jul 16 22:21:53.290652 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:53.290625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"801e1d03-9641-4637-a49b-41b35f8e0b9c","Type":"ContainerStarted","Data":"d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac"}
Jul 16 22:21:53.290787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:53.290675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"801e1d03-9641-4637-a49b-41b35f8e0b9c","Type":"ContainerStarted","Data":"54db1a05329b1e74f7235f3f717bc01c826f7e84a5f1eea3b289d82bd94e04f7"}
Jul 16 22:21:53.314875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:53.314804    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=4.314788882 podStartE2EDuration="4.314788882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:53.312615381 +0000 UTC m=+1924.291964680" watchObservedRunningTime="2026-07-16 22:21:53.314788882 +0000 UTC m=+1924.294138181"
Jul 16 22:21:53.336717 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:53.336659    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=4.336642877 podStartE2EDuration="4.336642877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:53.332659692 +0000 UTC m=+1924.312008991" watchObservedRunningTime="2026-07-16 22:21:53.336642877 +0000 UTC m=+1924.315992175"
Jul 16 22:21:54.251031 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:21:54.250979    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:22:00.602512 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:00.602447    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:22:00.613673 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:00.613617    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:22:02.250875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:02.250827    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:22:02.252834 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:02.252778    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:22:10.602562 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:10.602504    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:22:10.612641 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:10.612587    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:22:12.251970 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:12.251924    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:22:20.602775 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:20.602724    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:22:20.613444 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:20.613397    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:22:22.251936 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:22.251883    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:22:30.603128 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:30.603079    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:22:30.612749 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:30.612710    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:22:32.252212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:32.252158    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:22:40.603036 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:40.602935    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:22:40.612928 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:40.612877    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:22:42.251880 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:42.251834    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:22:50.602380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:50.602333    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:22:50.613564 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:50.613521    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:22:52.252064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:22:52.252024    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:00.602483 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:00.602437    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:23:00.613120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:00.613082    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:23:02.251181 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:02.251134    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:10.608490 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:10.608440    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:23:10.613009 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:10.612979    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:23:12.251630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:12.251586    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:20.602469 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:20.602379    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 16 22:23:20.612801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:20.612755    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 16 22:23:22.251599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:22.251543    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:30.614203 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:30.614161    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:23:30.626419 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:30.626391    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:23:30.629610 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:30.629590    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:23:30.637558 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:30.637532    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:23:32.251589 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:32.251534    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:42.251795 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:42.251744    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:52.251386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:52.251334    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:23:54.078948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:54.078908    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"]
Jul 16 22:23:54.084135 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:23:54.084107    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"]
Jul 16 22:24:02.252127 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:02.252075    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:24:09.105504 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:09.105451    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" containerID="cri-o://d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919" gracePeriod=45
Jul 16 22:24:09.108067 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:09.108014    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main" containerID="cri-o://fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d" gracePeriod=45
Jul 16 22:24:10.432026 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.431855    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="llm-d-routing-sidecar" containerID="cri-o://cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d" gracePeriod=44
Jul 16 22:24:10.602295 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.602242    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 16 22:24:10.604188 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.604145    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 16 22:24:10.605272 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.605255    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:24:10.716699 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.716669    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:24:10.729622 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.729596    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-dshm\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.729780 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.729652    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kserve-provision-location\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.729780 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.729721    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-home\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.729780 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.729754    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tls-certs\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.729969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.729797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-model-cache\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.729969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.729950    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59r94\" (UniqueName: \"kubernetes.io/projected/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kube-api-access-59r94\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.730088 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.730013    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tmp-dir\") pod \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\" (UID: \"f7bb386b-a783-41bc-8da7-8ef16018f6a3\") "
Jul 16 22:24:10.730088 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.730062    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.730579 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.730554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-home" (OuterVolumeSpecName: "home") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.730743 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.730579    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.731965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.731941    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-dshm" (OuterVolumeSpecName: "dshm") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.732231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.732211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:24:10.732511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.732495    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kube-api-access-59r94" (OuterVolumeSpecName: "kube-api-access-59r94") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "kube-api-access-59r94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:24:10.745825 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.745777    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.800927 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.800879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7bb386b-a783-41bc-8da7-8ef16018f6a3" (UID: "f7bb386b-a783-41bc-8da7-8ef16018f6a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.831693 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831656    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-model-cache\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.831903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831766    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tls-certs\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.831903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bm89\" (UniqueName: \"kubernetes.io/projected/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kube-api-access-4bm89\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.831903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831850    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kserve-provision-location\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.831903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831891    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-home\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.832141 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831952    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-dshm\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.832141 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.831978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tmp-dir\") pod \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\" (UID: \"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91\") "
Jul 16 22:24:10.832141 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832007    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.832318 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832299    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832325    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59r94\" (UniqueName: \"kubernetes.io/projected/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kube-api-access-59r94\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832338    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832351    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832361    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832373    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832640 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832384    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bb386b-a783-41bc-8da7-8ef16018f6a3-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.832640 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.832554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-home" (OuterVolumeSpecName: "home") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.834669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.834641    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-dshm" (OuterVolumeSpecName: "dshm") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.834782 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.834660    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kube-api-access-4bm89" (OuterVolumeSpecName: "kube-api-access-4bm89") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "kube-api-access-4bm89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:24:10.836395 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.836371    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:24:10.845828 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.845778    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.892971 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.892922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" (UID: "d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:10.933054 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.933018    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.933054 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.933052    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.933054 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.933063    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.933380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.933071    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.933380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.933081    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bm89\" (UniqueName: \"kubernetes.io/projected/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kube-api-access-4bm89\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.933380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:10.933089    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:24:11.199651 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.199612    2557 generic.go:356] "Generic (PLEG): container finished" podID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerID="fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d" exitCode=0
Jul 16 22:24:11.199875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.199680    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"
Jul 16 22:24:11.199875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.199687    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" event={"ID":"f7bb386b-a783-41bc-8da7-8ef16018f6a3","Type":"ContainerDied","Data":"fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d"}
Jul 16 22:24:11.199875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.199766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq" event={"ID":"f7bb386b-a783-41bc-8da7-8ef16018f6a3","Type":"ContainerDied","Data":"90494f01200221e98bdf7c8b5720d4d11f68a19318ea386421565ced2b0afabf"}
Jul 16 22:24:11.199875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.199799    2557 scope.go:117] "RemoveContainer" containerID="fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d"
Jul 16 22:24:11.202073 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.202049    2557 generic.go:356] "Generic (PLEG): container finished" podID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerID="d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919" exitCode=0
Jul 16 22:24:11.202073 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.202070    2557 generic.go:356] "Generic (PLEG): container finished" podID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerID="cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d" exitCode=0
Jul 16 22:24:11.202287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.202134    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"
Jul 16 22:24:11.202287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.202151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerDied","Data":"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"}
Jul 16 22:24:11.202287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.202234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerDied","Data":"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"}
Jul 16 22:24:11.202287 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.202263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" event={"ID":"d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91","Type":"ContainerDied","Data":"e9b7b1c7c1bf211679a511b6bbf41f77a69f3d0a8be763d1cdaee60bb0cf4bc0"}
Jul 16 22:24:11.211538 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.211516    2557 scope.go:117] "RemoveContainer" containerID="cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52"
Jul 16 22:24:11.222252 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.222232    2557 scope.go:117] "RemoveContainer" containerID="fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d"
Jul 16 22:24:11.222574 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:24:11.222554    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d\": container with ID starting with fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d not found: ID does not exist" containerID="fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d"
Jul 16 22:24:11.222630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.222584    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d"} err="failed to get container status \"fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d\": rpc error: code = NotFound desc = could not find container \"fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d\": container with ID starting with fd2b666166dcaadac4bc9b99a59ab6b1b59bcbc88aca356bff0a8aba94675d9d not found: ID does not exist"
Jul 16 22:24:11.222630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.222602    2557 scope.go:117] "RemoveContainer" containerID="cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52"
Jul 16 22:24:11.222875 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:24:11.222859    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52\": container with ID starting with cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52 not found: ID does not exist" containerID="cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52"
Jul 16 22:24:11.222932 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.222878    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52"} err="failed to get container status \"cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52\": rpc error: code = NotFound desc = could not find container \"cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52\": container with ID starting with cb5370e10941c58d7eef09549111d92f3ec895d0a810c222abd262bcaec4bf52 not found: ID does not exist"
Jul 16 22:24:11.222932 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.222890    2557 scope.go:117] "RemoveContainer" containerID="d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"
Jul 16 22:24:11.228374 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.228349    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"]
Jul 16 22:24:11.232584 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.232561    2557 scope.go:117] "RemoveContainer" containerID="4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"
Jul 16 22:24:11.232701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.232681    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bwl2gq"]
Jul 16 22:24:11.241659 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.241634    2557 scope.go:117] "RemoveContainer" containerID="cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"
Jul 16 22:24:11.246017 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.245984    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"]
Jul 16 22:24:11.250376 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.250343    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk"]
Jul 16 22:24:11.251676 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.251659    2557 scope.go:117] "RemoveContainer" containerID="d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"
Jul 16 22:24:11.252022 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:24:11.251994    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919\": container with ID starting with d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919 not found: ID does not exist" containerID="d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"
Jul 16 22:24:11.252118 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252034    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"} err="failed to get container status \"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919\": rpc error: code = NotFound desc = could not find container \"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919\": container with ID starting with d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919 not found: ID does not exist"
Jul 16 22:24:11.252118 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252058    2557 scope.go:117] "RemoveContainer" containerID="4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"
Jul 16 22:24:11.252357 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:24:11.252340    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a\": container with ID starting with 4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a not found: ID does not exist" containerID="4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"
Jul 16 22:24:11.252396 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252363    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"} err="failed to get container status \"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a\": rpc error: code = NotFound desc = could not find container \"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a\": container with ID starting with 4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a not found: ID does not exist"
Jul 16 22:24:11.252396 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252378    2557 scope.go:117] "RemoveContainer" containerID="cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"
Jul 16 22:24:11.252644 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:24:11.252622    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d\": container with ID starting with cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d not found: ID does not exist" containerID="cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"
Jul 16 22:24:11.252713 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252650    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"} err="failed to get container status \"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d\": rpc error: code = NotFound desc = could not find container \"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d\": container with ID starting with cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d not found: ID does not exist"
Jul 16 22:24:11.252713 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252669    2557 scope.go:117] "RemoveContainer" containerID="d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"
Jul 16 22:24:11.252924 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252905    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919"} err="failed to get container status \"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919\": rpc error: code = NotFound desc = could not find container \"d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919\": container with ID starting with d13be710aa150810cbb318807d41e398f8e9443c954caca8061aa00f2d8ae919 not found: ID does not exist"
Jul 16 22:24:11.252990 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.252927    2557 scope.go:117] "RemoveContainer" containerID="4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"
Jul 16 22:24:11.253161 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.253134    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a"} err="failed to get container status \"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a\": rpc error: code = NotFound desc = could not find container \"4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a\": container with ID starting with 4ebbf2552efe72fd1afdd1e6c581ec4a15dd23e7730c653c79268171ce88ff0a not found: ID does not exist"
Jul 16 22:24:11.253301 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.253164    2557 scope.go:117] "RemoveContainer" containerID="cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"
Jul 16 22:24:11.253460 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.253440    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d"} err="failed to get container status \"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d\": rpc error: code = NotFound desc = could not find container \"cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d\": container with ID starting with cd9b32c98c8faf07a103ef4a44a84ad345816e939d7f97fb32a797a613f5a41d not found: ID does not exist"
Jul 16 22:24:11.562703 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.562616    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" path="/var/lib/kubelet/pods/d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91/volumes"
Jul 16 22:24:11.563105 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:11.563091    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" path="/var/lib/kubelet/pods/f7bb386b-a783-41bc-8da7-8ef16018f6a3/volumes"
Jul 16 22:24:12.252048 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:12.252007    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:24:15.615738 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:15.615679    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6d4d8bcdbfhfsgk" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Jul 16 22:24:22.251944 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:22.251894    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:24:23.126024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.125990    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"]
Jul 16 22:24:23.126398 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126386    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main"
Jul 16 22:24:23.126452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126401    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main"
Jul 16 22:24:23.126452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126414    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="storage-initializer"
Jul 16 22:24:23.126452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126420    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="storage-initializer"
Jul 16 22:24:23.126452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126437    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="llm-d-routing-sidecar"
Jul 16 22:24:23.126452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126442    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="llm-d-routing-sidecar"
Jul 16 22:24:23.126452 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126451    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="storage-initializer"
Jul 16 22:24:23.126636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126460    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="storage-initializer"
Jul 16 22:24:23.126636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126473    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main"
Jul 16 22:24:23.126636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126478    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main"
Jul 16 22:24:23.126636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126538    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="llm-d-routing-sidecar"
Jul 16 22:24:23.126636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126549    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bb386b-a783-41bc-8da7-8ef16018f6a3" containerName="main"
Jul 16 22:24:23.126636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.126557    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fa2ad0-8e98-40f5-9af2-09f2cd5c7c91" containerName="main"
Jul 16 22:24:23.130956 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.130923    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.134341 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.134316    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:24:23.135375 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.135177    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-wgkzl\""
Jul 16 22:24:23.135375 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.135255    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 16 22:24:23.145365 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.145341    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"]
Jul 16 22:24:23.147664 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.147640    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"]
Jul 16 22:24:23.152571 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.152543    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.168161 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.168131    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"]
Jul 16 22:24:23.261967 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.261917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.261967 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.261965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.261994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-dshm\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-home\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff47x\" (UniqueName: \"kubernetes.io/projected/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kube-api-access-ff47x\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262311    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262363    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262391    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v764\" (UniqueName: \"kubernetes.io/projected/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kube-api-access-7v764\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.262472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.262412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.363386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363394    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-home\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff47x\" (UniqueName: \"kubernetes.io/projected/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kube-api-access-ff47x\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.363597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v764\" (UniqueName: \"kubernetes.io/projected/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kube-api-access-7v764\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363736    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-dshm\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363770    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.364005 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.363903    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-home\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.364088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.364404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.364113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.364173    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.364228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.364404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.364371    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.364633 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.364590    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.366394 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.366363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.366530 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.366478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-dshm\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.366701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.366676    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.366947 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.366929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.378285 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.378170    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff47x\" (UniqueName: \"kubernetes.io/projected/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kube-api-access-ff47x\") pod \"custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.383466 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.383433    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v764\" (UniqueName: \"kubernetes.io/projected/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kube-api-access-7v764\") pod \"custom-route-timeout-pd-test-kserve-7886454c6-qxk94\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.443950 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.443907    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:23.463215 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.462589    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:23.655709 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.655678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"]
Jul 16 22:24:23.656386 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:24:23.656351    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c5bbbb_afa7_4da3_8abf_a48138cfa933.slice/crio-7a5e322d95a7625949b2eff9134a6b0896088242db6304cbb5a4b1bbdf717475 WatchSource:0}: Error finding container 7a5e322d95a7625949b2eff9134a6b0896088242db6304cbb5a4b1bbdf717475: Status 404 returned error can't find the container with id 7a5e322d95a7625949b2eff9134a6b0896088242db6304cbb5a4b1bbdf717475
Jul 16 22:24:23.676298 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:23.676273    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"]
Jul 16 22:24:23.679389 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:24:23.679356    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dad25a_e62f_45f7_bb46_5cd86dce7ae8.slice/crio-f6caa376a4008235e4580d7c2327fe7372d38fba1d369b2c61d4767188aba5e1 WatchSource:0}: Error finding container f6caa376a4008235e4580d7c2327fe7372d38fba1d369b2c61d4767188aba5e1: Status 404 returned error can't find the container with id f6caa376a4008235e4580d7c2327fe7372d38fba1d369b2c61d4767188aba5e1
Jul 16 22:24:24.281776 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:24.281720    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" event={"ID":"17dad25a-e62f-45f7-bb46-5cd86dce7ae8","Type":"ContainerStarted","Data":"ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd"}
Jul 16 22:24:24.282276 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:24.281792    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" event={"ID":"17dad25a-e62f-45f7-bb46-5cd86dce7ae8","Type":"ContainerStarted","Data":"f6caa376a4008235e4580d7c2327fe7372d38fba1d369b2c61d4767188aba5e1"}
Jul 16 22:24:24.283472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:24.283442    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerStarted","Data":"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"}
Jul 16 22:24:24.283631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:24.283507    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:24.283631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:24.283531    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerStarted","Data":"7a5e322d95a7625949b2eff9134a6b0896088242db6304cbb5a4b1bbdf717475"}
Jul 16 22:24:25.291804 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:25.291753    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerStarted","Data":"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"}
Jul 16 22:24:27.305668 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:27.305575    2557 generic.go:356] "Generic (PLEG): container finished" podID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerID="ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd" exitCode=0
Jul 16 22:24:27.305668 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:27.305625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" event={"ID":"17dad25a-e62f-45f7-bb46-5cd86dce7ae8","Type":"ContainerDied","Data":"ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd"}
Jul 16 22:24:28.314105 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:28.314065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" event={"ID":"17dad25a-e62f-45f7-bb46-5cd86dce7ae8","Type":"ContainerStarted","Data":"da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c"}
Jul 16 22:24:28.316074 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:28.316044    2557 generic.go:356] "Generic (PLEG): container finished" podID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerID="cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a" exitCode=0
Jul 16 22:24:28.316194 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:28.316110    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerDied","Data":"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"}
Jul 16 22:24:28.359889 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:28.359827    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podStartSLOduration=2.069123192 podStartE2EDuration="5.359796002s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.29067281s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:28.357734061 +0000 UTC m=+2079.337083383" watchObservedRunningTime="2026-07-16 22:24:28.359796002 +0000 UTC m=+2079.339145298"
Jul 16 22:24:29.326184 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:29.326147    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerStarted","Data":"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"}
Jul 16 22:24:29.355439 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:29.355366    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podStartSLOduration=2.856316991 podStartE2EDuration="6.355343588s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.499026597s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:29.350960976 +0000 UTC m=+2080.330310275" watchObservedRunningTime="2026-07-16 22:24:29.355343588 +0000 UTC m=+2080.334692888"
Jul 16 22:24:32.251412 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:32.251344    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:24:33.444911 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.444864    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:33.445444 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.444943    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:33.447024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.446667    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:24:33.459342 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.459308    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:24:33.463298 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.463266    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:33.463505 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.463329    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:24:33.464909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:33.464877    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:24:42.262372 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:42.262336    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:24:42.272286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:42.272232    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:24:43.445331 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:43.445280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:24:43.463437 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:43.463398    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:24:49.715288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:49.715250    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:24:49.715756 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:49.715357    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:24:49.720168 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:49.720144    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:24:49.720647 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:49.720627    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:24:53.445043 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:53.444996    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:24:53.463260 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:24:53.463221    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:25:03.445110 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:03.445049    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:25:03.463693 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:03.463648    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:25:13.444737 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:13.444671    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:25:13.464082 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:13.464039    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:25:23.444718 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:23.444670    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:25:23.463833 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:23.463783    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:25:33.445376 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:33.445327    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:25:33.464271 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:33.464228    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:25:43.444598 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:43.444556    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:25:43.463589 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:43.463544    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:25:53.444746 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:53.444701    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:25:53.464211 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:25:53.464153    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:26:03.444951 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:03.444900    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:26:03.463551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:03.463507    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:26:13.444435 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:13.444381    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:26:13.464044 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:13.464005    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:26:23.445210 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:23.445148    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:26:23.464000 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:23.463961    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:26:33.444637 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:33.444586    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:26:33.463510 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:33.463457    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:26:43.444800 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:43.444735    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:26:43.464067 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:43.464027    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:26:53.444604 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:53.444559    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:26:53.463756 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:26:53.463714    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:27:03.444842 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:03.444781    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Jul 16 22:27:03.463985 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:03.463935    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 16 22:27:13.454614 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:13.454538    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:27:13.467447 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:13.467419    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:27:13.474288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:13.474262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:27:13.482086 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:13.482052    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:27:19.911864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:19.911806    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"]
Jul 16 22:27:19.914523 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:19.914486    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"]
Jul 16 22:27:25.122711 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:25.122664    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 22:27:34.939020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:34.938947    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main" containerID="cri-o://0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a" gracePeriod=45
Jul 16 22:27:34.939708 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:34.939520    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main" containerID="cri-o://da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c" gracePeriod=45
Jul 16 22:27:35.751466 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:35.751411    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="llm-d-routing-sidecar" containerID="cri-o://a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e" gracePeriod=45
Jul 16 22:27:35.902349 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:35.902317    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:27:36.013987 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.013962    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:27:36.047762 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.047730    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tmp-dir\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.047962 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.047797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-model-cache\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.047962 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.047837    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tls-certs\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.047962 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.047866    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kserve-provision-location\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.047962 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.047911    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff47x\" (UniqueName: \"kubernetes.io/projected/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kube-api-access-ff47x\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.047962 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.047962    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-dshm\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.048247 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.048013    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-home\") pod \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\" (UID: \"17dad25a-e62f-45f7-bb46-5cd86dce7ae8\") "
Jul 16 22:27:36.048247 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.048052    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-model-cache" (OuterVolumeSpecName: "model-cache") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.048397 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.048328    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.049280 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.049244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-home" (OuterVolumeSpecName: "home") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.050309 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.050280    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:36.050629 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.050597    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-dshm" (OuterVolumeSpecName: "dshm") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.050749 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.050620    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kube-api-access-ff47x" (OuterVolumeSpecName: "kube-api-access-ff47x") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "kube-api-access-ff47x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:36.061288 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.061251    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.093411 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.093366    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17dad25a-e62f-45f7-bb46-5cd86dce7ae8" (UID: "17dad25a-e62f-45f7-bb46-5cd86dce7ae8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.148584 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148552    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-model-cache\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.148725 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148602    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-home\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.148725 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tmp-dir\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.148725 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148665    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v764\" (UniqueName: \"kubernetes.io/projected/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kube-api-access-7v764\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.148725 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148684    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-dshm\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.148725 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148711    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kserve-provision-location\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.149062 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148761    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tls-certs\") pod \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\" (UID: \"13c5bbbb-afa7-4da3-8abf-a48138cfa933\") "
Jul 16 22:27:36.149062 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.148878    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.149243 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149214    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149314 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149252    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149314 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149265    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149314 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149280    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff47x\" (UniqueName: \"kubernetes.io/projected/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-kube-api-access-ff47x\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149314 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149294    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149314 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149306    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149736 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149317    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dad25a-e62f-45f7-bb46-5cd86dce7ae8-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.149736 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.149431    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-home" (OuterVolumeSpecName: "home") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.151385 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.151358    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-dshm" (OuterVolumeSpecName: "dshm") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.151593 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.151578    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:36.151774 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.151756    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kube-api-access-7v764" (OuterVolumeSpecName: "kube-api-access-7v764") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "kube-api-access-7v764". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:36.164423 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.164392    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.216165 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.216122    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13c5bbbb-afa7-4da3-8abf-a48138cfa933" (UID: "13c5bbbb-afa7-4da3-8abf-a48138cfa933"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:36.250104 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.250071    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.250198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.250111    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.250198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.250122    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.250198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.250131    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.250198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.250139    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c5bbbb-afa7-4da3-8abf-a48138cfa933-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.250198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.250147    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v764\" (UniqueName: \"kubernetes.io/projected/13c5bbbb-afa7-4da3-8abf-a48138cfa933-kube-api-access-7v764\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.535340 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.535297    2557 generic.go:356] "Generic (PLEG): container finished" podID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerID="da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c" exitCode=0
Jul 16 22:27:36.535549 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.535353    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"
Jul 16 22:27:36.535549 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.535375    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" event={"ID":"17dad25a-e62f-45f7-bb46-5cd86dce7ae8","Type":"ContainerDied","Data":"da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c"}
Jul 16 22:27:36.535549 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.535431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb" event={"ID":"17dad25a-e62f-45f7-bb46-5cd86dce7ae8","Type":"ContainerDied","Data":"f6caa376a4008235e4580d7c2327fe7372d38fba1d369b2c61d4767188aba5e1"}
Jul 16 22:27:36.535549 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.535453    2557 scope.go:117] "RemoveContainer" containerID="da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c"
Jul 16 22:27:36.537837 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.537792    2557 generic.go:356] "Generic (PLEG): container finished" podID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerID="0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a" exitCode=0
Jul 16 22:27:36.537837 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.537834    2557 generic.go:356] "Generic (PLEG): container finished" podID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerID="a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e" exitCode=0
Jul 16 22:27:36.538033 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.537865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerDied","Data":"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"}
Jul 16 22:27:36.538033 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.537907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerDied","Data":"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"}
Jul 16 22:27:36.538033 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.537938    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"
Jul 16 22:27:36.538033 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.537953    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94" event={"ID":"13c5bbbb-afa7-4da3-8abf-a48138cfa933","Type":"ContainerDied","Data":"7a5e322d95a7625949b2eff9134a6b0896088242db6304cbb5a4b1bbdf717475"}
Jul 16 22:27:36.546967 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.546942    2557 scope.go:117] "RemoveContainer" containerID="ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd"
Jul 16 22:27:36.556571 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.556550    2557 scope.go:117] "RemoveContainer" containerID="da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c"
Jul 16 22:27:36.556911 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:36.556881    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c\": container with ID starting with da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c not found: ID does not exist" containerID="da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c"
Jul 16 22:27:36.557031 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.556916    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c"} err="failed to get container status \"da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c\": rpc error: code = NotFound desc = could not find container \"da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c\": container with ID starting with da99820c49ba7b90b8097fe5a365da05c1b0a83e1ffb8bf7c153fa2cb1b1078c not found: ID does not exist"
Jul 16 22:27:36.557031 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.556933    2557 scope.go:117] "RemoveContainer" containerID="ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd"
Jul 16 22:27:36.557258 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:36.557229    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd\": container with ID starting with ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd not found: ID does not exist" containerID="ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd"
Jul 16 22:27:36.557367 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.557265    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd"} err="failed to get container status \"ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd\": rpc error: code = NotFound desc = could not find container \"ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd\": container with ID starting with ea2132e46d11ce028a24d7abf7654740e0b18bc6479c340c824cdc9e8387fcdd not found: ID does not exist"
Jul 16 22:27:36.557367 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.557281    2557 scope.go:117] "RemoveContainer" containerID="0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"
Jul 16 22:27:36.560581 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.560560    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"]
Jul 16 22:27:36.565586 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.565561    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-fbb965dc8-4nrzb"]
Jul 16 22:27:36.566729 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.566715    2557 scope.go:117] "RemoveContainer" containerID="cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"
Jul 16 22:27:36.576566 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.576545    2557 scope.go:117] "RemoveContainer" containerID="a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"
Jul 16 22:27:36.580831 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.580786    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"]
Jul 16 22:27:36.584948 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.584922    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7886454c6-qxk94"]
Jul 16 22:27:36.589208 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.589190    2557 scope.go:117] "RemoveContainer" containerID="0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"
Jul 16 22:27:36.589530 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:36.589509    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a\": container with ID starting with 0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a not found: ID does not exist" containerID="0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"
Jul 16 22:27:36.589658 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.589538    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"} err="failed to get container status \"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a\": rpc error: code = NotFound desc = could not find container \"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a\": container with ID starting with 0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a not found: ID does not exist"
Jul 16 22:27:36.589658 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.589557    2557 scope.go:117] "RemoveContainer" containerID="cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"
Jul 16 22:27:36.589868 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:36.589846    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a\": container with ID starting with cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a not found: ID does not exist" containerID="cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"
Jul 16 22:27:36.589973 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.589869    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"} err="failed to get container status \"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a\": rpc error: code = NotFound desc = could not find container \"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a\": container with ID starting with cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a not found: ID does not exist"
Jul 16 22:27:36.589973 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.589882    2557 scope.go:117] "RemoveContainer" containerID="a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"
Jul 16 22:27:36.590141 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:36.590125    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e\": container with ID starting with a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e not found: ID does not exist" containerID="a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"
Jul 16 22:27:36.590196 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590145    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"} err="failed to get container status \"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e\": rpc error: code = NotFound desc = could not find container \"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e\": container with ID starting with a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e not found: ID does not exist"
Jul 16 22:27:36.590196 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590159    2557 scope.go:117] "RemoveContainer" containerID="0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"
Jul 16 22:27:36.590392 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590369    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a"} err="failed to get container status \"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a\": rpc error: code = NotFound desc = could not find container \"0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a\": container with ID starting with 0e546fa667601f7db09d1b523800b07a14267b97bb01fc0cae24f422f9bd403a not found: ID does not exist"
Jul 16 22:27:36.590511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590394    2557 scope.go:117] "RemoveContainer" containerID="cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"
Jul 16 22:27:36.590679 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590662    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a"} err="failed to get container status \"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a\": rpc error: code = NotFound desc = could not find container \"cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a\": container with ID starting with cdcdfc9a826e10cb94bd361277afcd4407050dfd36b687314120a2747dd5314a not found: ID does not exist"
Jul 16 22:27:36.590749 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590682    2557 scope.go:117] "RemoveContainer" containerID="a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"
Jul 16 22:27:36.590970 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:36.590952    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e"} err="failed to get container status \"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e\": rpc error: code = NotFound desc = could not find container \"a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e\": container with ID starting with a6c2cc6f0f5ba6ea34167764dbb19ee6f1ce5be98306041a3f8691ee6a0e053e not found: ID does not exist"
Jul 16 22:27:37.564685 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:37.564644    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" path="/var/lib/kubelet/pods/13c5bbbb-afa7-4da3-8abf-a48138cfa933/volumes"
Jul 16 22:27:37.565278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:37.565261    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" path="/var/lib/kubelet/pods/17dad25a-e62f-45f7-bb46-5cd86dce7ae8/volumes"
Jul 16 22:27:40.154285 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:40.154246    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" containerID="cri-o://57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2" gracePeriod=45
Jul 16 22:27:40.263742 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:40.263701    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 22:27:40.961713 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:40.961682    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:27:41.098352 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098314    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tmp-dir\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.098547 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098395    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-dshm\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.098547 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098420    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-home\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.098648 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098586    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.098648 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098622    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqtvd\" (UniqueName: \"kubernetes.io/projected/783c323c-d2d9-46d8-88ea-3c7afe0a238c-kube-api-access-pqtvd\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.098752 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098650    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tls-certs\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.098752 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.098682    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-model-cache\") pod \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\" (UID: \"783c323c-d2d9-46d8-88ea-3c7afe0a238c\") "
Jul 16 22:27:41.099662 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.099624    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-model-cache" (OuterVolumeSpecName: "model-cache") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:41.099831 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.099699    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-home" (OuterVolumeSpecName: "home") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:41.101151 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.101128    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-dshm" (OuterVolumeSpecName: "dshm") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:41.101566 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.101541    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783c323c-d2d9-46d8-88ea-3c7afe0a238c-kube-api-access-pqtvd" (OuterVolumeSpecName: "kube-api-access-pqtvd") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "kube-api-access-pqtvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:41.101727 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.101707    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:41.102535 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.102515    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8" (OuterVolumeSpecName: "kserve-pvc-source") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 22:27:41.112182 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.112145    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "783c323c-d2d9-46d8-88ea-3c7afe0a238c" (UID: "783c323c-d2d9-46d8-88ea-3c7afe0a238c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:41.200991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.200890    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqtvd\" (UniqueName: \"kubernetes.io/projected/783c323c-d2d9-46d8-88ea-3c7afe0a238c-kube-api-access-pqtvd\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.200991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.200933    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.200991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.200949    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.200991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.200962    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.200991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.200974    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.200991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.200984    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c323c-d2d9-46d8-88ea-3c7afe0a238c-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.567991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.567893    2557 generic.go:356] "Generic (PLEG): container finished" podID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerID="57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2" exitCode=0
Jul 16 22:27:41.567991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.567934    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 22:27:41.567991 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.567973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"783c323c-d2d9-46d8-88ea-3c7afe0a238c","Type":"ContainerDied","Data":"57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2"}
Jul 16 22:27:41.568222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.568028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"783c323c-d2d9-46d8-88ea-3c7afe0a238c","Type":"ContainerDied","Data":"44b395bf9083e2431f66267ba2c4528bb748ab4967cda2a28248e444ea7e82f4"}
Jul 16 22:27:41.568222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.568053    2557 scope.go:117] "RemoveContainer" containerID="57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2"
Jul 16 22:27:41.585329 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.585304    2557 scope.go:117] "RemoveContainer" containerID="57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2"
Jul 16 22:27:41.585697 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:41.585674    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2\": container with ID starting with 57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2 not found: ID does not exist" containerID="57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2"
Jul 16 22:27:41.585745 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.585706    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2"} err="failed to get container status \"57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2\": rpc error: code = NotFound desc = could not find container \"57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2\": container with ID starting with 57f0602642eb8845bc2ad613ba43d991a47a20288cee6627e7bd9ffba67d6fc2 not found: ID does not exist"
Jul 16 22:27:41.598806 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.598776    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 22:27:41.602903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.602874    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 22:27:41.618097 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:41.618070    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 22:27:43.563039 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:43.563002    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" path="/var/lib/kubelet/pods/783c323c-d2d9-46d8-88ea-3c7afe0a238c/volumes"
Jul 16 22:27:49.351046 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351006    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"]
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351388    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="storage-initializer"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351400    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="storage-initializer"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351416    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="llm-d-routing-sidecar"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351421    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="llm-d-routing-sidecar"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351437    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="storage-initializer"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351443    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="storage-initializer"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351449    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351454    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351461    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351466    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351472    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main"
Jul 16 22:27:49.351488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351477    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main"
Jul 16 22:27:49.351876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351536    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dad25a-e62f-45f7-bb46-5cd86dce7ae8" containerName="main"
Jul 16 22:27:49.351876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351545    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="783c323c-d2d9-46d8-88ea-3c7afe0a238c" containerName="main"
Jul 16 22:27:49.351876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351553    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="llm-d-routing-sidecar"
Jul 16 22:27:49.351876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.351559    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c5bbbb-afa7-4da3-8abf-a48138cfa933" containerName="main"
Jul 16 22:27:49.356740 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.356713    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.359460 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.359430    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-5m6wg\""
Jul 16 22:27:49.359621 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.359467    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 16 22:27:49.359758 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.359738    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:27:49.362009 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.361989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.362132 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.362015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-model-cache\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.362132 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.362050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pt5m\" (UniqueName: \"kubernetes.io/projected/ffcdebcc-451a-4020-ad72-0b150832144e-kube-api-access-7pt5m\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.362262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.362133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcdebcc-451a-4020-ad72-0b150832144e-tls-certs\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.362262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.362183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-home\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.362262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.362240    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-dshm\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.362429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.362270    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.376834 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.376785    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"]
Jul 16 22:27:49.380556 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.380531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.396495 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.396453    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"]
Jul 16 22:27:49.409116 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.409078    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"]
Jul 16 22:27:49.463260 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-home\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463260 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-dshm\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463515 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463515 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463515 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2r4\" (UniqueName: \"kubernetes.io/projected/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kube-api-access-nm2r4\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463515 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463515 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463513    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-model-cache\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-home\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-home\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pt5m\" (UniqueName: \"kubernetes.io/projected/ffcdebcc-451a-4020-ad72-0b150832144e-kube-api-access-7pt5m\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463747    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.463769 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcdebcc-451a-4020-ad72-0b150832144e-tls-certs\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.464125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.463797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-model-cache\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.465801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.465779    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-dshm\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.467411 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.467394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 16 22:27:49.476833 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.476789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcdebcc-451a-4020-ad72-0b150832144e-tls-certs\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.476953 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.476932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pt5m\" (UniqueName: \"kubernetes.io/projected/ffcdebcc-451a-4020-ad72-0b150832144e-kube-api-access-7pt5m\") pod \"router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.564245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2r4\" (UniqueName: \"kubernetes.io/projected/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kube-api-access-nm2r4\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-home\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564717 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564452    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.564846 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564803    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.565018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.565018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.564953    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-home\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.565018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.565004    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.566588 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.566566    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.567143 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.567123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.574317 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.574292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2r4\" (UniqueName: \"kubernetes.io/projected/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kube-api-access-nm2r4\") pod \"router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.671086 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.671053    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-5m6wg\""
Jul 16 22:27:49.678368 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.678345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:49.691921 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.691895    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:49.845516 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.845487    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"]
Jul 16 22:27:49.846775 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:27:49.846747    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcdebcc_451a_4020_ad72_0b150832144e.slice/crio-3430bd4b978ea909f12c19806a8881ebaa2725584a1c49ab8fa2038096021e02 WatchSource:0}: Error finding container 3430bd4b978ea909f12c19806a8881ebaa2725584a1c49ab8fa2038096021e02: Status 404 returned error can't find the container with id 3430bd4b978ea909f12c19806a8881ebaa2725584a1c49ab8fa2038096021e02
Jul 16 22:27:49.873903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.873877    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"]
Jul 16 22:27:49.880930 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:27:49.880899    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fff4fa4_17a9_47a9_b0a0_58eb42fd8762.slice/crio-0b7a134ef04813744ef50afe991b431cf882604f3dcbffe75c04573a8cfff3d2 WatchSource:0}: Error finding container 0b7a134ef04813744ef50afe991b431cf882604f3dcbffe75c04573a8cfff3d2: Status 404 returned error can't find the container with id 0b7a134ef04813744ef50afe991b431cf882604f3dcbffe75c04573a8cfff3d2
Jul 16 22:27:49.887935 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:49.887911    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:27:50.626607 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:50.626570    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" event={"ID":"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762","Type":"ContainerStarted","Data":"1d508201648bd9b60465db3970041439dda17f4bc7d8a41e60b5226bb71be688"}
Jul 16 22:27:50.627081 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:50.626641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" event={"ID":"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762","Type":"ContainerStarted","Data":"0b7a134ef04813744ef50afe991b431cf882604f3dcbffe75c04573a8cfff3d2"}
Jul 16 22:27:50.628439 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:50.628411    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerStarted","Data":"242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716"}
Jul 16 22:27:50.628560 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:50.628466    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerStarted","Data":"3430bd4b978ea909f12c19806a8881ebaa2725584a1c49ab8fa2038096021e02"}
Jul 16 22:27:50.628560 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:50.628495    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:51.636381 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:51.636347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerStarted","Data":"227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa"}
Jul 16 22:27:53.650067 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:53.650019    2557 generic.go:356] "Generic (PLEG): container finished" podID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerID="1d508201648bd9b60465db3970041439dda17f4bc7d8a41e60b5226bb71be688" exitCode=0
Jul 16 22:27:53.650067 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:53.650065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" event={"ID":"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762","Type":"ContainerDied","Data":"1d508201648bd9b60465db3970041439dda17f4bc7d8a41e60b5226bb71be688"}
Jul 16 22:27:54.659600 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:54.659557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" event={"ID":"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762","Type":"ContainerStarted","Data":"afb71427645eb77f8aa70e824ae0c5f79d5326f969f327179307f5bf4d663289"}
Jul 16 22:27:54.661643 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:54.661614    2557 generic.go:356] "Generic (PLEG): container finished" podID="ffcdebcc-451a-4020-ad72-0b150832144e" containerID="227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa" exitCode=0
Jul 16 22:27:54.661832 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:54.661649    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerDied","Data":"227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa"}
Jul 16 22:27:54.685444 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:54.685385    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podStartSLOduration=2.192391009 podStartE2EDuration="5.685363635s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.492972626s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:54.683333008 +0000 UTC m=+2285.662682332" watchObservedRunningTime="2026-07-16 22:27:54.685363635 +0000 UTC m=+2285.664712935"
Jul 16 22:27:55.670616 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:55.670580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerStarted","Data":"8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd"}
Jul 16 22:27:55.704090 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:55.704024    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podStartSLOduration=3.269804646 podStartE2EDuration="6.704003328s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.434198682s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:55.698837894 +0000 UTC m=+2286.678187193" watchObservedRunningTime="2026-07-16 22:27:55.704003328 +0000 UTC m=+2286.683352629"
Jul 16 22:27:56.641893 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:56.641847    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="801e1d03-9641-4637-a49b-41b35f8e0b9c" containerName="main" containerID="cri-o://d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac" gracePeriod=45
Jul 16 22:27:57.425282 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.425236    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:27:57.457836 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.457723    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-home\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.457836 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.457803    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-tmp-dir\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.458060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.457876    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-model-cache\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.458060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.457985    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.458060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.458025    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjdtg\" (UniqueName: \"kubernetes.io/projected/801e1d03-9641-4637-a49b-41b35f8e0b9c-kube-api-access-tjdtg\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.458225 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.458083    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801e1d03-9641-4637-a49b-41b35f8e0b9c-tls-certs\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.458225 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.458142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-dshm\") pod \"801e1d03-9641-4637-a49b-41b35f8e0b9c\" (UID: \"801e1d03-9641-4637-a49b-41b35f8e0b9c\") "
Jul 16 22:27:57.458592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.458403    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:57.458743 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.458715    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-home" (OuterVolumeSpecName: "home") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:57.472543 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.472507    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8" (OuterVolumeSpecName: "kserve-pvc-source") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 22:27:57.480750 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.480221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801e1d03-9641-4637-a49b-41b35f8e0b9c-kube-api-access-tjdtg" (OuterVolumeSpecName: "kube-api-access-tjdtg") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "kube-api-access-tjdtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:57.480750 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.480381    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801e1d03-9641-4637-a49b-41b35f8e0b9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:57.482641 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.482607    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-dshm" (OuterVolumeSpecName: "dshm") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:57.496198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.496146    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "801e1d03-9641-4637-a49b-41b35f8e0b9c" (UID: "801e1d03-9641-4637-a49b-41b35f8e0b9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:57.559341 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559305    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801e1d03-9641-4637-a49b-41b35f8e0b9c-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:57.559341 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559334    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:57.559341 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559343    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:57.559594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559371    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:57.559594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559380    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801e1d03-9641-4637-a49b-41b35f8e0b9c-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:57.559594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559410    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-136-80.ec2.internal\" "
Jul 16 22:27:57.559594 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.559424    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdtg\" (UniqueName: \"kubernetes.io/projected/801e1d03-9641-4637-a49b-41b35f8e0b9c-kube-api-access-tjdtg\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:27:57.589702 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.589677    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8") on node "ip-10-0-136-80.ec2.internal"
Jul 16 22:27:57.660183 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.660138    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"csi-d72f08b41b54ae6d564237c1c81095268ae69f02eb78be48ca7868e9c1c2304b\""
Jul 16 22:27:57.683180 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.683145    2557 generic.go:356] "Generic (PLEG): container finished" podID="801e1d03-9641-4637-a49b-41b35f8e0b9c" containerID="d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac" exitCode=0
Jul 16 22:27:57.683368 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.683225    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 22:27:57.683368 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.683233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"801e1d03-9641-4637-a49b-41b35f8e0b9c","Type":"ContainerDied","Data":"d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac"}
Jul 16 22:27:57.683368 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.683295    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"801e1d03-9641-4637-a49b-41b35f8e0b9c","Type":"ContainerDied","Data":"54db1a05329b1e74f7235f3f717bc01c826f7e84a5f1eea3b289d82bd94e04f7"}
Jul 16 22:27:57.683368 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.683326    2557 scope.go:117] "RemoveContainer" containerID="d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac"
Jul 16 22:27:57.693700 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.693673    2557 scope.go:117] "RemoveContainer" containerID="d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac"
Jul 16 22:27:57.694025 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:27:57.694004    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac\": container with ID starting with d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac not found: ID does not exist" containerID="d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac"
Jul 16 22:27:57.694124 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.694033    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac"} err="failed to get container status \"d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac\": rpc error: code = NotFound desc = could not find container \"d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac\": container with ID starting with d4299dfb7a76c9ab112b2ec6ab0e1fbc214b45627c6635db497ea48d9270e2ac not found: ID does not exist"
Jul 16 22:27:57.734480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.734371    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 22:27:57.747581 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:57.747549    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 22:27:59.563401 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.563365    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801e1d03-9641-4637-a49b-41b35f8e0b9c" path="/var/lib/kubelet/pods/801e1d03-9641-4637-a49b-41b35f8e0b9c/volumes"
Jul 16 22:27:59.678766 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.678716    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:59.679001 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.678788    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:27:59.680260 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.680224    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:27:59.692245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.692218    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:59.692511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.692494    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:27:59.693894 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.693860    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:27:59.699873 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:27:59.699848    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:28:09.679575 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:09.679528    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:28:09.693060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:09.693011    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:28:09.987888 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:09.987785    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"]
Jul 16 22:28:09.988222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:09.988206    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801e1d03-9641-4637-a49b-41b35f8e0b9c" containerName="main"
Jul 16 22:28:09.988222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:09.988222    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="801e1d03-9641-4637-a49b-41b35f8e0b9c" containerName="main"
Jul 16 22:28:09.988355 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:09.988298    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="801e1d03-9641-4637-a49b-41b35f8e0b9c" containerName="main"
Jul 16 22:28:10.013097 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.013065    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"]
Jul 16 22:28:10.013284 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.013187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.016183 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.016152    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 16 22:28:10.078869 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.078796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-model-cache\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.079057 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.078901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0044145f-189a-4468-a301-9c6b01766749-tls-certs\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.079057 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.078955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-home\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.079057 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.079015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-dshm\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.079057 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.079048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84sz\" (UniqueName: \"kubernetes.io/projected/0044145f-189a-4468-a301-9c6b01766749-kube-api-access-n84sz\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.079296 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.079102    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-tmp-dir\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.079296 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.079157    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-kserve-provision-location\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.179856 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.179796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-model-cache\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.179856 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.179861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0044145f-189a-4468-a301-9c6b01766749-tls-certs\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180129 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.179897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-home\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180129 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.179934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-dshm\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180129 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.179968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84sz\" (UniqueName: \"kubernetes.io/projected/0044145f-189a-4468-a301-9c6b01766749-kube-api-access-n84sz\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180129 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.180007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-tmp-dir\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180129 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.180062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-kserve-provision-location\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180496 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.180471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-tmp-dir\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.180483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-kserve-provision-location\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.180578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-home\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.180715 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.180601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-model-cache\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.182455 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.182430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-dshm\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.183136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.183111    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0044145f-189a-4468-a301-9c6b01766749-tls-certs\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.191405 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.191343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84sz\" (UniqueName: \"kubernetes.io/projected/0044145f-189a-4468-a301-9c6b01766749-kube-api-access-n84sz\") pod \"stop-feature-test-kserve-779bc4c586-hpj62\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.328075 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.327979    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:10.519232 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.519202    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"]
Jul 16 22:28:10.521780 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:28:10.521750    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0044145f_189a_4468_a301_9c6b01766749.slice/crio-516b4a9bea1ee1622fa2cc67499d94a0f2dae3edbf17792b2d1415d18b87f081 WatchSource:0}: Error finding container 516b4a9bea1ee1622fa2cc67499d94a0f2dae3edbf17792b2d1415d18b87f081: Status 404 returned error can't find the container with id 516b4a9bea1ee1622fa2cc67499d94a0f2dae3edbf17792b2d1415d18b87f081
Jul 16 22:28:10.770266 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.770224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" event={"ID":"0044145f-189a-4468-a301-9c6b01766749","Type":"ContainerStarted","Data":"9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067"}
Jul 16 22:28:10.770788 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:10.770294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" event={"ID":"0044145f-189a-4468-a301-9c6b01766749","Type":"ContainerStarted","Data":"516b4a9bea1ee1622fa2cc67499d94a0f2dae3edbf17792b2d1415d18b87f081"}
Jul 16 22:28:11.580480 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:11.580431    2557 pod_container_manager_linux.go:217] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod783c323c-d2d9-46d8-88ea-3c7afe0a238c"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod783c323c-d2d9-46d8-88ea-3c7afe0a238c] : Timed out while waiting for systemd to remove kubepods-burstable-pod783c323c_d2d9_46d8_88ea_3c7afe0a238c.slice"
Jul 16 22:28:14.808715 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:14.808674    2557 generic.go:356] "Generic (PLEG): container finished" podID="0044145f-189a-4468-a301-9c6b01766749" containerID="9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067" exitCode=0
Jul 16 22:28:14.809243 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:14.808745    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" event={"ID":"0044145f-189a-4468-a301-9c6b01766749","Type":"ContainerDied","Data":"9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067"}
Jul 16 22:28:15.819115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:15.819071    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" event={"ID":"0044145f-189a-4468-a301-9c6b01766749","Type":"ContainerStarted","Data":"97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c"}
Jul 16 22:28:16.329002 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:16.328968    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:19.679055 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:19.679006    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:28:19.692969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:19.692922    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:28:20.329170 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:20.329129    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:28:20.331249 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:20.331211    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:28:29.679557 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:29.679505    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:28:29.692470 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:29.692430    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:28:30.329424 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:30.329373    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:28:39.679610 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:39.679559    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:28:39.693078 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:39.693035    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:28:40.329147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:40.329052    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:28:49.679891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:49.679468    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:28:49.693043 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:49.693000    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:28:50.328833 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:50.328773    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:28:59.679776 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:59.679335    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:28:59.693195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:28:59.693146    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:29:00.329511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:00.329461    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:29:09.679447 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:09.679392    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:29:09.692723 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:09.692670    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:29:10.329072 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:10.329023    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:29:19.679076 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:19.679026    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:29:19.692437 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:19.692391    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:29:20.328969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:20.328927    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:29:29.679216 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:29.679149    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:29:29.693154 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:29.693110    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:29:30.329110 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:30.329068    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:29:39.678898 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:39.678846    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:29:39.692457 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:39.692416    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:29:40.329024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:40.328981    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:29:49.679481 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:49.679433    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:29:49.692212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:49.692171    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:29:49.766531 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:49.766501    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:29:49.767606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:49.767588    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:29:49.771053 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:49.771035    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:29:49.772085 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:49.772066    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:29:50.328609 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:50.328558    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:29:59.679836 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:59.679776    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:29:59.693055 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:29:59.693011    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:30:00.328476 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:00.328427    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:30:09.679768 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:09.679714    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:30:09.692377 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:09.692333    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:30:10.328789 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:10.328699    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:30:19.679711 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:19.679646    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:30:19.692777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:19.692740    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:30:20.328587 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:20.328544    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:30:29.679043 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:29.678997    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:30:29.693228 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:29.693180    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:30:30.328794 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:30.328739    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:30:39.678894 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:39.678809    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:30:39.692628 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:39.692578    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:30:40.329180 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:40.329122    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:30:49.679042 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:49.678995    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:30:49.693214 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:49.693175    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:30:50.329194 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:50.329151    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:30:59.679077 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:59.679014    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Jul 16 22:30:59.693030 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:30:59.692987    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:31:00.329127 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:00.329089    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:31:09.688714 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:09.688671    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:31:09.702506 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:09.702478    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:31:09.707047 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:09.707021    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:31:09.714873 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:09.714849    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:31:09.727716 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:09.727662    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podStartSLOduration=177.450693113 podStartE2EDuration="3m0.727641973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.27694886s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:15.86151747 +0000 UTC m=+2306.840866773" watchObservedRunningTime="2026-07-16 22:31:09.727641973 +0000 UTC m=+2480.706991271"
Jul 16 22:31:10.328866 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:10.328793    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:31:20.329404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:20.329355    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:31:24.229120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:24.229075    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"]
Jul 16 22:31:24.254761 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:24.254726    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"]
Jul 16 22:31:30.342068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:30.342035    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:31:30.350627 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:30.350602    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:31:31.649061 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:31.649024    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"]
Jul 16 22:31:39.257422 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:39.257381    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main" containerID="cri-o://8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd" gracePeriod=45
Jul 16 22:31:39.278735 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:39.278693    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" containerID="cri-o://afb71427645eb77f8aa70e824ae0c5f79d5326f969f327179307f5bf4d663289" gracePeriod=45
Jul 16 22:31:39.703626 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:39.703589    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 16 22:31:40.085481 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.085442    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="llm-d-routing-sidecar" containerID="cri-o://242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716" gracePeriod=45
Jul 16 22:31:40.100486 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.100456    2557 generic.go:356] "Generic (PLEG): container finished" podID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerID="afb71427645eb77f8aa70e824ae0c5f79d5326f969f327179307f5bf4d663289" exitCode=0
Jul 16 22:31:40.100651 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.100533    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" event={"ID":"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762","Type":"ContainerDied","Data":"afb71427645eb77f8aa70e824ae0c5f79d5326f969f327179307f5bf4d663289"}
Jul 16 22:31:40.102648 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.102626    2557 generic.go:356] "Generic (PLEG): container finished" podID="ffcdebcc-451a-4020-ad72-0b150832144e" containerID="8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd" exitCode=0
Jul 16 22:31:40.102711 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.102695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerDied","Data":"8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd"}
Jul 16 22:31:40.243167 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.243102    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:31:40.321307 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321270    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tls-certs\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.321731 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321333    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-home\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.321731 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321366    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kserve-provision-location\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.321731 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321384    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tmp-dir\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.321731 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321414    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-model-cache\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.321731 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321454    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm2r4\" (UniqueName: \"kubernetes.io/projected/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kube-api-access-nm2r4\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.321731 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.321500    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-dshm\") pod \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\" (UID: \"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762\") "
Jul 16 22:31:40.322153 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.322030    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-home" (OuterVolumeSpecName: "home") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.322212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.322183    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.325424 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.325390    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:31:40.325551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.325523    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-dshm" (OuterVolumeSpecName: "dshm") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.325597 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.325541    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kube-api-access-nm2r4" (OuterVolumeSpecName: "kube-api-access-nm2r4") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "kube-api-access-nm2r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:40.335897 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.335860    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.356185 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.356159    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:31:40.395467 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.395424    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" (UID: "3fff4fa4-17a9-47a9-b0a0-58eb42fd8762"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.422764 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422722    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.422764 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422757    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.422764 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422766    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.422764 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422775    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.423068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422784    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.423068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422792    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm2r4\" (UniqueName: \"kubernetes.io/projected/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-kube-api-access-nm2r4\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.423068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.422799    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.523199 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523098    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pt5m\" (UniqueName: \"kubernetes.io/projected/ffcdebcc-451a-4020-ad72-0b150832144e-kube-api-access-7pt5m\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523199 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523164    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-tmp-dir\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523199 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523197    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-home\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523494 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523360    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-model-cache\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523494 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523405    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-kserve-provision-location\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523494 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523457    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-dshm\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523494 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523489    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcdebcc-451a-4020-ad72-0b150832144e-tls-certs\") pod \"ffcdebcc-451a-4020-ad72-0b150832144e\" (UID: \"ffcdebcc-451a-4020-ad72-0b150832144e\") "
Jul 16 22:31:40.523942 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.523579    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.524090 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.524039    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-home" (OuterVolumeSpecName: "home") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.525635 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.525602    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcdebcc-451a-4020-ad72-0b150832144e-kube-api-access-7pt5m" (OuterVolumeSpecName: "kube-api-access-7pt5m") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "kube-api-access-7pt5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:40.525736 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.525675    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-dshm" (OuterVolumeSpecName: "dshm") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.526147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.526115    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcdebcc-451a-4020-ad72-0b150832144e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:31:40.536147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.536112    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.589914 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.589863    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffcdebcc-451a-4020-ad72-0b150832144e" (UID: "ffcdebcc-451a-4020-ad72-0b150832144e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:40.625020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.624988    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pt5m\" (UniqueName: \"kubernetes.io/projected/ffcdebcc-451a-4020-ad72-0b150832144e-kube-api-access-7pt5m\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.625020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.625017    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.625020 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.625027    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.625231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.625035    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.625231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.625049    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.625231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.625061    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcdebcc-451a-4020-ad72-0b150832144e-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.625231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:40.625068    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcdebcc-451a-4020-ad72-0b150832144e-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:41.109079 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.109054    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"
Jul 16 22:31:41.109280 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.109053    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd" event={"ID":"3fff4fa4-17a9-47a9-b0a0-58eb42fd8762","Type":"ContainerDied","Data":"0b7a134ef04813744ef50afe991b431cf882604f3dcbffe75c04573a8cfff3d2"}
Jul 16 22:31:41.109280 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.109208    2557 scope.go:117] "RemoveContainer" containerID="afb71427645eb77f8aa70e824ae0c5f79d5326f969f327179307f5bf4d663289"
Jul 16 22:31:41.111160 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.111130    2557 generic.go:356] "Generic (PLEG): container finished" podID="ffcdebcc-451a-4020-ad72-0b150832144e" containerID="242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716" exitCode=0
Jul 16 22:31:41.111293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.111187    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"
Jul 16 22:31:41.111293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.111214    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerDied","Data":"242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716"}
Jul 16 22:31:41.111293 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.111259    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79" event={"ID":"ffcdebcc-451a-4020-ad72-0b150832144e","Type":"ContainerDied","Data":"3430bd4b978ea909f12c19806a8881ebaa2725584a1c49ab8fa2038096021e02"}
Jul 16 22:31:41.119337 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.119308    2557 scope.go:117] "RemoveContainer" containerID="1d508201648bd9b60465db3970041439dda17f4bc7d8a41e60b5226bb71be688"
Jul 16 22:31:41.127971 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.127950    2557 scope.go:117] "RemoveContainer" containerID="8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd"
Jul 16 22:31:41.133778 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.133753    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"]
Jul 16 22:31:41.137147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.137112    2557 scope.go:117] "RemoveContainer" containerID="227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa"
Jul 16 22:31:41.137391 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.137370    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6ddb4b6c98-wphcd"]
Jul 16 22:31:41.145338 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.145321    2557 scope.go:117] "RemoveContainer" containerID="242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716"
Jul 16 22:31:41.151561 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.151449    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"]
Jul 16 22:31:41.155266 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.155243    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5ccd7c9ddd-sbl79"]
Jul 16 22:31:41.155585 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.155567    2557 scope.go:117] "RemoveContainer" containerID="8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd"
Jul 16 22:31:41.155898 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:31:41.155880    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd\": container with ID starting with 8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd not found: ID does not exist" containerID="8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd"
Jul 16 22:31:41.155956 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.155907    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd"} err="failed to get container status \"8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd\": rpc error: code = NotFound desc = could not find container \"8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd\": container with ID starting with 8b0b94e572390a611c57567fb5b45ed49dd987ee855365f18b14bf88a3e90bbd not found: ID does not exist"
Jul 16 22:31:41.155956 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.155924    2557 scope.go:117] "RemoveContainer" containerID="227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa"
Jul 16 22:31:41.156180 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:31:41.156161    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa\": container with ID starting with 227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa not found: ID does not exist" containerID="227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa"
Jul 16 22:31:41.156226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.156186    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa"} err="failed to get container status \"227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa\": rpc error: code = NotFound desc = could not find container \"227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa\": container with ID starting with 227319c821e7ac485b6881fdae00a637df5c456eb8881c870a60c92f077b3eaa not found: ID does not exist"
Jul 16 22:31:41.156226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.156203    2557 scope.go:117] "RemoveContainer" containerID="242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716"
Jul 16 22:31:41.156419 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:31:41.156405    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716\": container with ID starting with 242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716 not found: ID does not exist" containerID="242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716"
Jul 16 22:31:41.156460 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.156423    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716"} err="failed to get container status \"242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716\": rpc error: code = NotFound desc = could not find container \"242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716\": container with ID starting with 242feef092bbb815a3707308b403d26cf07d04fe8d9f8fb48ab6dfac9cfaf716 not found: ID does not exist"
Jul 16 22:31:41.562056 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.562023    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" path="/var/lib/kubelet/pods/3fff4fa4-17a9-47a9-b0a0-58eb42fd8762/volumes"
Jul 16 22:31:41.562448 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:41.562435    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" path="/var/lib/kubelet/pods/ffcdebcc-451a-4020-ad72-0b150832144e/volumes"
Jul 16 22:31:45.192353 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192312    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"]
Jul 16 22:31:45.192890 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192872    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="llm-d-routing-sidecar"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192893    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="llm-d-routing-sidecar"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192905    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="storage-initializer"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192914    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="storage-initializer"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192928    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192936    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192964    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="storage-initializer"
Jul 16 22:31:45.192984 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192973    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="storage-initializer"
Jul 16 22:31:45.193333 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192986    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main"
Jul 16 22:31:45.193333 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.192997    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main"
Jul 16 22:31:45.193333 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.193102    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="llm-d-routing-sidecar"
Jul 16 22:31:45.193333 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.193116    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fff4fa4-17a9-47a9-b0a0-58eb42fd8762" containerName="main"
Jul 16 22:31:45.193333 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.193125    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcdebcc-451a-4020-ad72-0b150832144e" containerName="main"
Jul 16 22:31:45.198424 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.198402    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.206212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.206037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"]
Jul 16 22:31:45.372573 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372534    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-kserve-provision-location\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.372573 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372571    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea941e-5362-47d1-8aad-08793e54120d-tls-certs\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.372876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-model-cache\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.372876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-home\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.372876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-tmp-dir\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.372876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bmr\" (UniqueName: \"kubernetes.io/projected/b8ea941e-5362-47d1-8aad-08793e54120d-kube-api-access-v7bmr\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.372876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.372852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-dshm\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474261 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-tmp-dir\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474261 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474217    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bmr\" (UniqueName: \"kubernetes.io/projected/b8ea941e-5362-47d1-8aad-08793e54120d-kube-api-access-v7bmr\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474261 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474241    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-dshm\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-kserve-provision-location\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea941e-5362-47d1-8aad-08793e54120d-tls-certs\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-model-cache\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-home\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-tmp-dir\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474741    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-kserve-provision-location\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474770    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-model-cache\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.474927 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.474857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-home\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.476688 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.476671    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-dshm\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.476896 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.476875    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea941e-5362-47d1-8aad-08793e54120d-tls-certs\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.482336 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.482311    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bmr\" (UniqueName: \"kubernetes.io/projected/b8ea941e-5362-47d1-8aad-08793e54120d-kube-api-access-v7bmr\") pod \"stop-feature-test-kserve-779bc4c586-xrcs9\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.509801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.509777    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:45.677349 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:45.677309    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"]
Jul 16 22:31:45.679707 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:31:45.679678    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ea941e_5362_47d1_8aad_08793e54120d.slice/crio-d67fff78c67a90d38d592651be6904a072808aa7ffdb219ed30679aa581e5bdb WatchSource:0}: Error finding container d67fff78c67a90d38d592651be6904a072808aa7ffdb219ed30679aa581e5bdb: Status 404 returned error can't find the container with id d67fff78c67a90d38d592651be6904a072808aa7ffdb219ed30679aa581e5bdb
Jul 16 22:31:46.142464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:46.142426    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" event={"ID":"b8ea941e-5362-47d1-8aad-08793e54120d","Type":"ContainerStarted","Data":"c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352"}
Jul 16 22:31:46.142646 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:46.142483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" event={"ID":"b8ea941e-5362-47d1-8aad-08793e54120d","Type":"ContainerStarted","Data":"d67fff78c67a90d38d592651be6904a072808aa7ffdb219ed30679aa581e5bdb"}
Jul 16 22:31:47.075993 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.075949    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" containerID="cri-o://97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c" gracePeriod=45
Jul 16 22:31:47.343044 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.342929    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 22:31:47.865421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.865390    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:31:47.899927 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.899833    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-model-cache\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.899927 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.899897    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0044145f-189a-4468-a301-9c6b01766749-tls-certs\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.899927 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.899924    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-dshm\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.900251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.899954    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-home\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.900251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.900018    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n84sz\" (UniqueName: \"kubernetes.io/projected/0044145f-189a-4468-a301-9c6b01766749-kube-api-access-n84sz\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.900251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.900047    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-model-cache" (OuterVolumeSpecName: "model-cache") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:47.900251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.900136    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-kserve-provision-location\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.900251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.900164    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-tmp-dir\") pod \"0044145f-189a-4468-a301-9c6b01766749\" (UID: \"0044145f-189a-4468-a301-9c6b01766749\") "
Jul 16 22:31:47.900531 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.900492    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:47.901551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.901508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-home" (OuterVolumeSpecName: "home") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:47.902885 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.902845    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0044145f-189a-4468-a301-9c6b01766749-kube-api-access-n84sz" (OuterVolumeSpecName: "kube-api-access-n84sz") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "kube-api-access-n84sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:47.904859 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.904782    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0044145f-189a-4468-a301-9c6b01766749-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:31:47.906935 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.906905    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-dshm" (OuterVolumeSpecName: "dshm") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:47.919778 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.919710    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:47.969198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:47.969156    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0044145f-189a-4468-a301-9c6b01766749" (UID: "0044145f-189a-4468-a301-9c6b01766749"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:48.001316 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.001267    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84sz\" (UniqueName: \"kubernetes.io/projected/0044145f-189a-4468-a301-9c6b01766749-kube-api-access-n84sz\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:48.001316 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.001317    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:48.001617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.001333    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:48.001617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.001347    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0044145f-189a-4468-a301-9c6b01766749-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:48.001617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.001362    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:48.001617 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.001376    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0044145f-189a-4468-a301-9c6b01766749-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:31:48.160278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.159974    2557 generic.go:356] "Generic (PLEG): container finished" podID="0044145f-189a-4468-a301-9c6b01766749" containerID="97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c" exitCode=0
Jul 16 22:31:48.160278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.160046    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"
Jul 16 22:31:48.160278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.160063    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" event={"ID":"0044145f-189a-4468-a301-9c6b01766749","Type":"ContainerDied","Data":"97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c"}
Jul 16 22:31:48.160278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.160136    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62" event={"ID":"0044145f-189a-4468-a301-9c6b01766749","Type":"ContainerDied","Data":"516b4a9bea1ee1622fa2cc67499d94a0f2dae3edbf17792b2d1415d18b87f081"}
Jul 16 22:31:48.160278 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.160169    2557 scope.go:117] "RemoveContainer" containerID="97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c"
Jul 16 22:31:48.181194 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.181161    2557 scope.go:117] "RemoveContainer" containerID="9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067"
Jul 16 22:31:48.190623 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.190592    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"]
Jul 16 22:31:48.192743 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.192723    2557 scope.go:117] "RemoveContainer" containerID="97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c"
Jul 16 22:31:48.193136 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:31:48.193103    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c\": container with ID starting with 97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c not found: ID does not exist" containerID="97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c"
Jul 16 22:31:48.193317 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.193146    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c"} err="failed to get container status \"97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c\": rpc error: code = NotFound desc = could not find container \"97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c\": container with ID starting with 97225b50d3beb25468305092d44dddd65a9dfb5dbde895364051883b7aee653c not found: ID does not exist"
Jul 16 22:31:48.193317 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.193173    2557 scope.go:117] "RemoveContainer" containerID="9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067"
Jul 16 22:31:48.193576 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:31:48.193554    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067\": container with ID starting with 9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067 not found: ID does not exist" containerID="9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067"
Jul 16 22:31:48.193576 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.193585    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067"} err="failed to get container status \"9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067\": rpc error: code = NotFound desc = could not find container \"9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067\": container with ID starting with 9b456f8c20f92d7c53a0a60e4f3ab79fdab2fa98734b4f69aaa7cd6522a53067 not found: ID does not exist"
Jul 16 22:31:48.196006 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:48.195986    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-hpj62"]
Jul 16 22:31:49.168411 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:49.168375    2557 generic.go:356] "Generic (PLEG): container finished" podID="b8ea941e-5362-47d1-8aad-08793e54120d" containerID="c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352" exitCode=0
Jul 16 22:31:49.168783 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:49.168436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" event={"ID":"b8ea941e-5362-47d1-8aad-08793e54120d","Type":"ContainerDied","Data":"c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352"}
Jul 16 22:31:49.563523 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:49.563445    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0044145f-189a-4468-a301-9c6b01766749" path="/var/lib/kubelet/pods/0044145f-189a-4468-a301-9c6b01766749/volumes"
Jul 16 22:31:50.176782 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:50.176690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" event={"ID":"b8ea941e-5362-47d1-8aad-08793e54120d","Type":"ContainerStarted","Data":"c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2"}
Jul 16 22:31:50.203079 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:50.203009    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podStartSLOduration=2.343816298 podStartE2EDuration="5.202984402s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.859168104s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:50.199977294 +0000 UTC m=+2521.179326597" watchObservedRunningTime="2026-07-16 22:31:50.202984402 +0000 UTC m=+2521.182333702"
Jul 16 22:31:50.510380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:50.510295    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:55.510615 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:55.510568    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:31:55.512109 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:55.512080    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:31:56.671228 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.671192    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"]
Jul 16 22:31:56.671727 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.671703    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="storage-initializer"
Jul 16 22:31:56.671792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.671744    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="storage-initializer"
Jul 16 22:31:56.671792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.671774    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main"
Jul 16 22:31:56.671792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.671783    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main"
Jul 16 22:31:56.671925 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.671917    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0044145f-189a-4468-a301-9c6b01766749" containerName="main"
Jul 16 22:31:56.675663 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.675638    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.678504 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.678478    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 16 22:31:56.690698 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.690664    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"]
Jul 16 22:31:56.779631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.779868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.779868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.779868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44drj\" (UniqueName: \"kubernetes.io/projected/547e67e9-5b3f-4b91-a368-4901641f7223-kube-api-access-44drj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.779868 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547e67e9-5b3f-4b91-a368-4901641f7223-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.780070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.780070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.779943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.880803 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.880760    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881019 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.880844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881019 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.880871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44drj\" (UniqueName: \"kubernetes.io/projected/547e67e9-5b3f-4b91-a368-4901641f7223-kube-api-access-44drj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881019 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.880903    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547e67e9-5b3f-4b91-a368-4901641f7223-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881041    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881226 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881197    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881459 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881539 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881517    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.881763 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.881537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.883348 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.883317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.883593 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.883575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547e67e9-5b3f-4b91-a368-4901641f7223-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.889890 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.889867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44drj\" (UniqueName: \"kubernetes.io/projected/547e67e9-5b3f-4b91-a368-4901641f7223-kube-api-access-44drj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:56.988152 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:56.988073    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:31:57.161496 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:57.161468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"]
Jul 16 22:31:57.163591 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:31:57.163560    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547e67e9_5b3f_4b91_a368_4901641f7223.slice/crio-ac528d4c4f8020930ce9160a7558eb718a26eb5da0e9d9b427d937d867e14e80 WatchSource:0}: Error finding container ac528d4c4f8020930ce9160a7558eb718a26eb5da0e9d9b427d937d867e14e80: Status 404 returned error can't find the container with id ac528d4c4f8020930ce9160a7558eb718a26eb5da0e9d9b427d937d867e14e80
Jul 16 22:31:57.216873 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:57.216843    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" event={"ID":"547e67e9-5b3f-4b91-a368-4901641f7223","Type":"ContainerStarted","Data":"ac528d4c4f8020930ce9160a7558eb718a26eb5da0e9d9b427d937d867e14e80"}
Jul 16 22:31:58.233237 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:31:58.233112    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" event={"ID":"547e67e9-5b3f-4b91-a368-4901641f7223","Type":"ContainerStarted","Data":"bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3"}
Jul 16 22:32:01.255121 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:01.255085    2557 generic.go:356] "Generic (PLEG): container finished" podID="547e67e9-5b3f-4b91-a368-4901641f7223" containerID="bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3" exitCode=0
Jul 16 22:32:01.255611 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:01.255131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" event={"ID":"547e67e9-5b3f-4b91-a368-4901641f7223","Type":"ContainerDied","Data":"bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3"}
Jul 16 22:32:02.264802 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:02.264751    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" event={"ID":"547e67e9-5b3f-4b91-a368-4901641f7223","Type":"ContainerStarted","Data":"1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677"}
Jul 16 22:32:02.289483 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:02.289414    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podStartSLOduration=3.227105384 podStartE2EDuration="6.289393986s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.062288602s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:32:02.288450188 +0000 UTC m=+2533.267799489" watchObservedRunningTime="2026-07-16 22:32:02.289393986 +0000 UTC m=+2533.268743286"
Jul 16 22:32:02.988387 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:02.988349    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:32:05.510780 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:05.510725    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:32:06.988720 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:06.988686    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:32:06.990563 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:06.990534    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:32:15.510217 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:15.510174    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:32:16.989332 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:16.989280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:32:25.510348 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:25.510288    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:32:26.989601 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:26.989554    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:32:35.510469 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:35.510421    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:32:36.989172 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:36.989115    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:32:45.510380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:45.510338    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:32:46.988963 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:46.988907    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:32:55.510895 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:55.510835    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:32:56.989369 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:32:56.989312    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:33:05.510585 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:05.510528    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:33:06.989147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:06.989108    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:33:15.511206 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:15.511104    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:33:16.989385 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:16.989338    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:33:25.510212 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:25.510169    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:33:26.989251 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:26.989209    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:33:35.510418 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:35.510366    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:33:36.989068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:36.989017    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:33:45.510180 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:45.510139    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:33:46.988592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:46.988545    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:33:55.511277 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:55.511228    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:33:56.989476 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:33:56.989423    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:34:05.510832 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:05.510780    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:34:06.988757 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:06.988717    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:34:15.520075 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:15.520033    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:34:15.528129 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:15.528094    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:34:16.989494 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:16.989455    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:34:17.202382 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:17.202345    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"]
Jul 16 22:34:26.999100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:26.999065    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:34:27.006891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:27.006861    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:34:31.742101 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:31.742068    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"]
Jul 16 22:34:32.226387 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:32.226340    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" containerID="cri-o://c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2" gracePeriod=45
Jul 16 22:34:32.521879 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:32.521757    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 16 22:34:32.966465 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:32.966439    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:34:33.052227 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052195    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-dshm\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052227 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052231    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-home\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052277    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7bmr\" (UniqueName: \"kubernetes.io/projected/b8ea941e-5362-47d1-8aad-08793e54120d-kube-api-access-v7bmr\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052303    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea941e-5362-47d1-8aad-08793e54120d-tls-certs\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052330    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-model-cache\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052697 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-tmp-dir\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052697 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052569    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-kserve-provision-location\") pod \"b8ea941e-5362-47d1-8aad-08793e54120d\" (UID: \"b8ea941e-5362-47d1-8aad-08793e54120d\") "
Jul 16 22:34:33.052697 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052676    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:33.052973 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052953    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.053021 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.052961    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-home" (OuterVolumeSpecName: "home") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:33.054658 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.054619    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ea941e-5362-47d1-8aad-08793e54120d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:34:33.054913 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.054886    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-dshm" (OuterVolumeSpecName: "dshm") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:33.055060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.055038    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ea941e-5362-47d1-8aad-08793e54120d-kube-api-access-v7bmr" (OuterVolumeSpecName: "kube-api-access-v7bmr") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "kube-api-access-v7bmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:34:33.066358 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.066324    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:33.134636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.134544    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8ea941e-5362-47d1-8aad-08793e54120d" (UID: "b8ea941e-5362-47d1-8aad-08793e54120d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:33.153938 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.153900    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.153938 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.153936    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.154092 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.153947    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7bmr\" (UniqueName: \"kubernetes.io/projected/b8ea941e-5362-47d1-8aad-08793e54120d-kube-api-access-v7bmr\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.154092 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.153956    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea941e-5362-47d1-8aad-08793e54120d-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.154092 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.153966    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.154092 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.153974    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea941e-5362-47d1-8aad-08793e54120d-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.187551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.187510    2557 generic.go:356] "Generic (PLEG): container finished" podID="b8ea941e-5362-47d1-8aad-08793e54120d" containerID="c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2" exitCode=0
Jul 16 22:34:33.187730 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.187565    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"
Jul 16 22:34:33.187730 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.187599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" event={"ID":"b8ea941e-5362-47d1-8aad-08793e54120d","Type":"ContainerDied","Data":"c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2"}
Jul 16 22:34:33.187730 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.187658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9" event={"ID":"b8ea941e-5362-47d1-8aad-08793e54120d","Type":"ContainerDied","Data":"d67fff78c67a90d38d592651be6904a072808aa7ffdb219ed30679aa581e5bdb"}
Jul 16 22:34:33.187730 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.187675    2557 scope.go:117] "RemoveContainer" containerID="c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2"
Jul 16 22:34:33.198705 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.198676    2557 scope.go:117] "RemoveContainer" containerID="c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352"
Jul 16 22:34:33.208559 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.208531    2557 scope.go:117] "RemoveContainer" containerID="c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2"
Jul 16 22:34:33.208908 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:34:33.208880    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2\": container with ID starting with c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2 not found: ID does not exist" containerID="c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2"
Jul 16 22:34:33.209024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.208939    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2"} err="failed to get container status \"c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2\": rpc error: code = NotFound desc = could not find container \"c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2\": container with ID starting with c33e32f899089060b9ba37e6cb2aa985cc46f4d95896b6fe3a5ed018c2c9f1f2 not found: ID does not exist"
Jul 16 22:34:33.209024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.208965    2557 scope.go:117] "RemoveContainer" containerID="c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352"
Jul 16 22:34:33.209335 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:34:33.209263    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352\": container with ID starting with c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352 not found: ID does not exist" containerID="c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352"
Jul 16 22:34:33.209335 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.209302    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352"} err="failed to get container status \"c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352\": rpc error: code = NotFound desc = could not find container \"c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352\": container with ID starting with c6c1bd3f541015384ca7e9d71511ad22f5005e1e4960e08d7fc84bdacbf78352 not found: ID does not exist"
Jul 16 22:34:33.211655 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.211631    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"]
Jul 16 22:34:33.218624 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.218601    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-779bc4c586-xrcs9"]
Jul 16 22:34:33.562933 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:33.562886    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" path="/var/lib/kubelet/pods/b8ea941e-5362-47d1-8aad-08793e54120d/volumes"
Jul 16 22:34:46.766776 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:46.766704    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" containerID="cri-o://1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677" gracePeriod=45
Jul 16 22:34:47.000224 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.000171    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 16 22:34:47.528636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.528609    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:34:47.682057 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682023    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-home\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682088    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-dshm\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682119    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-kserve-provision-location\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682174    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-model-cache\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682262 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-tmp-dir\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682479 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682307    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547e67e9-5b3f-4b91-a368-4901641f7223-tls-certs\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682479 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682361    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44drj\" (UniqueName: \"kubernetes.io/projected/547e67e9-5b3f-4b91-a368-4901641f7223-kube-api-access-44drj\") pod \"547e67e9-5b3f-4b91-a368-4901641f7223\" (UID: \"547e67e9-5b3f-4b91-a368-4901641f7223\") "
Jul 16 22:34:47.682479 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682428    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-model-cache" (OuterVolumeSpecName: "model-cache") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:47.682704 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682688    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:47.682878 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.682851    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-home" (OuterVolumeSpecName: "home") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:47.685096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.685070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547e67e9-5b3f-4b91-a368-4901641f7223-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:34:47.685096 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.685080    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-dshm" (OuterVolumeSpecName: "dshm") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:47.685237 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.685218    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547e67e9-5b3f-4b91-a368-4901641f7223-kube-api-access-44drj" (OuterVolumeSpecName: "kube-api-access-44drj") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "kube-api-access-44drj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:34:47.704370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.704332    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:47.749902 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.749854    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "547e67e9-5b3f-4b91-a368-4901641f7223" (UID: "547e67e9-5b3f-4b91-a368-4901641f7223"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:47.784134 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.784090    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547e67e9-5b3f-4b91-a368-4901641f7223-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:47.784134 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.784134    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44drj\" (UniqueName: \"kubernetes.io/projected/547e67e9-5b3f-4b91-a368-4901641f7223-kube-api-access-44drj\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:47.784134 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.784143    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:47.784545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.784154    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:47.784545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.784162    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:47.784545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:47.784170    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547e67e9-5b3f-4b91-a368-4901641f7223-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:34:48.268119 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.268084    2557 generic.go:356] "Generic (PLEG): container finished" podID="547e67e9-5b3f-4b91-a368-4901641f7223" containerID="1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677" exitCode=0
Jul 16 22:34:48.268270 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.268128    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"
Jul 16 22:34:48.268270 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.268167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" event={"ID":"547e67e9-5b3f-4b91-a368-4901641f7223","Type":"ContainerDied","Data":"1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677"}
Jul 16 22:34:48.268270 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.268219    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h" event={"ID":"547e67e9-5b3f-4b91-a368-4901641f7223","Type":"ContainerDied","Data":"ac528d4c4f8020930ce9160a7558eb718a26eb5da0e9d9b427d937d867e14e80"}
Jul 16 22:34:48.268270 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.268241    2557 scope.go:117] "RemoveContainer" containerID="1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677"
Jul 16 22:34:48.280551 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.280530    2557 scope.go:117] "RemoveContainer" containerID="bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3"
Jul 16 22:34:48.289394 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.289369    2557 scope.go:117] "RemoveContainer" containerID="1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677"
Jul 16 22:34:48.289684 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:34:48.289664    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677\": container with ID starting with 1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677 not found: ID does not exist" containerID="1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677"
Jul 16 22:34:48.289737 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.289694    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677"} err="failed to get container status \"1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677\": rpc error: code = NotFound desc = could not find container \"1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677\": container with ID starting with 1ed9b405b385de665aaf207ae53f713e50199770aa1023253a930b6b9c8a7677 not found: ID does not exist"
Jul 16 22:34:48.289737 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.289713    2557 scope.go:117] "RemoveContainer" containerID="bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3"
Jul 16 22:34:48.290032 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:34:48.290007    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3\": container with ID starting with bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3 not found: ID does not exist" containerID="bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3"
Jul 16 22:34:48.290159 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.290038    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3"} err="failed to get container status \"bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3\": rpc error: code = NotFound desc = could not find container \"bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3\": container with ID starting with bdfccf2d56e3a2377dbc03322f0ddf404fbcda9f0eb7dc0f5b15425d29cbc6b3 not found: ID does not exist"
Jul 16 22:34:48.292573 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.292550    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"]
Jul 16 22:34:48.298074 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:48.298054    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb864fd8f-krp9h"]
Jul 16 22:34:49.563968 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:49.563937    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" path="/var/lib/kubelet/pods/547e67e9-5b3f-4b91-a368-4901641f7223/volumes"
Jul 16 22:34:49.803003 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:49.802976    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:34:49.805012 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:49.804990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:34:49.807470 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:49.807435    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:34:49.809630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:34:49.809610    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:35:00.319982 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.319947    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"]
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320320    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320330    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320341    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="storage-initializer"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320346    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="storage-initializer"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320364    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320370    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320377    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="storage-initializer"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320382    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="storage-initializer"
Jul 16 22:35:00.320440 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320440    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="547e67e9-5b3f-4b91-a368-4901641f7223" containerName="main"
Jul 16 22:35:00.320722 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.320448    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ea941e-5362-47d1-8aad-08793e54120d" containerName="main"
Jul 16 22:35:00.323404 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.323386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.326203 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.326178    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:35:00.326370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.326202    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 16 22:35:00.326370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.326214    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:35:00.326370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.326218    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:35:00.326370 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.326180    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:35:00.326545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.326534    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:35:00.340024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.340004    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"]
Jul 16 22:35:00.385632 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385606    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tls-certs\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-model-cache\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385726    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tmp-dir\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385915 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ftl\" (UniqueName: \"kubernetes.io/projected/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kube-api-access-v6ftl\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385915 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385915 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-home\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.385915 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.385884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-dshm\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486528 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-model-cache\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486528 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486533    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tmp-dir\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ftl\" (UniqueName: \"kubernetes.io/projected/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kube-api-access-v6ftl\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-home\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-dshm\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tls-certs\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.486964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.486771    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.487317 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.487139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-model-cache\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.487317 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.487178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.487388 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.487361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tmp-dir\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.487477 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.487451    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-home\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.487696 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.487675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.489883 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.489856    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-dshm\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.490146 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.490124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tls-certs\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.495132 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.495110    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ftl\" (UniqueName: \"kubernetes.io/projected/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kube-api-access-v6ftl\") pod \"lora-single-adapter-test-kserve-786667d575-d9ffw\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.634210 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.634115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:00.786614 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:00.786566    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"]
Jul 16 22:35:00.787896 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:35:00.787868    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b6f5f7_1408_43d2_aa3a_10f068a6bfa0.slice/crio-d964fc44066bb504164ce616cb234452d762c79745a463a0a8d206a95d341126 WatchSource:0}: Error finding container d964fc44066bb504164ce616cb234452d762c79745a463a0a8d206a95d341126: Status 404 returned error can't find the container with id d964fc44066bb504164ce616cb234452d762c79745a463a0a8d206a95d341126
Jul 16 22:35:01.341056 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:01.341013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" event={"ID":"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0","Type":"ContainerStarted","Data":"9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d"}
Jul 16 22:35:01.341545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:01.341076    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" event={"ID":"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0","Type":"ContainerStarted","Data":"d964fc44066bb504164ce616cb234452d762c79745a463a0a8d206a95d341126"}
Jul 16 22:35:04.362299 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:04.362202    2557 generic.go:356] "Generic (PLEG): container finished" podID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerID="9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d" exitCode=0
Jul 16 22:35:04.362299 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:04.362277    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" event={"ID":"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0","Type":"ContainerDied","Data":"9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d"}
Jul 16 22:35:05.369472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:05.369435    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" event={"ID":"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0","Type":"ContainerStarted","Data":"6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320"}
Jul 16 22:35:05.398691 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:05.398639    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podStartSLOduration=2.279562115 podStartE2EDuration="5.398625213s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.119063098s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:05.395766763 +0000 UTC m=+2716.375116062" watchObservedRunningTime="2026-07-16 22:35:05.398625213 +0000 UTC m=+2716.377974513"
Jul 16 22:35:05.635068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:05.634983    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:10.634507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:10.634468    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:35:10.636071 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:10.636038    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:35:12.810739 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.810700    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 22:35:12.833070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.833040    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.833515 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.833488    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 22:35:12.836267 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.836238    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-2fllx\""
Jul 16 22:35:12.836424 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.836270    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 16 22:35:12.908987 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.908959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.909189 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.908993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.909189 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.909025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25373d74-3d14-41e1-a3df-38d3c9d1723b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.909189 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.909043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.909189 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.909097    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.909189 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.909113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfdk9\" (UniqueName: \"kubernetes.io/projected/25373d74-3d14-41e1-a3df-38d3c9d1723b-kube-api-access-rfdk9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:12.909396 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:12.909195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.009864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.009795    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.009873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfdk9\" (UniqueName: \"kubernetes.io/projected/25373d74-3d14-41e1-a3df-38d3c9d1723b-kube-api-access-rfdk9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.009920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.009975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010000    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25373d74-3d14-41e1-a3df-38d3c9d1723b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010064 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010376 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010166    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010376 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010479 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010378    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.010479 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.010416    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.012324 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.012300    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.012512 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.012495    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25373d74-3d14-41e1-a3df-38d3c9d1723b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.019663 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.019642    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfdk9\" (UniqueName: \"kubernetes.io/projected/25373d74-3d14-41e1-a3df-38d3c9d1723b-kube-api-access-rfdk9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.147147 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.147109    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:13.394508 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.394473    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 22:35:13.395574 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:35:13.395547    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25373d74_3d14_41e1_a3df_38d3c9d1723b.slice/crio-98d8e58339cb651d59f7b31da9eea7fc2afc44843fd9a1a680a83f760955a030 WatchSource:0}: Error finding container 98d8e58339cb651d59f7b31da9eea7fc2afc44843fd9a1a680a83f760955a030: Status 404 returned error can't find the container with id 98d8e58339cb651d59f7b31da9eea7fc2afc44843fd9a1a680a83f760955a030
Jul 16 22:35:13.418285 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:13.418246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"25373d74-3d14-41e1-a3df-38d3c9d1723b","Type":"ContainerStarted","Data":"98d8e58339cb651d59f7b31da9eea7fc2afc44843fd9a1a680a83f760955a030"}
Jul 16 22:35:14.425879 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:14.425841    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"25373d74-3d14-41e1-a3df-38d3c9d1723b","Type":"ContainerStarted","Data":"b7f8624e6969d92fc14cab3638b230132abcf72205a0d9632e8cdcb62d42317d"}
Jul 16 22:35:17.446227 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:17.446191    2557 generic.go:356] "Generic (PLEG): container finished" podID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerID="b7f8624e6969d92fc14cab3638b230132abcf72205a0d9632e8cdcb62d42317d" exitCode=0
Jul 16 22:35:17.446739 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:17.446267    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"25373d74-3d14-41e1-a3df-38d3c9d1723b","Type":"ContainerDied","Data":"b7f8624e6969d92fc14cab3638b230132abcf72205a0d9632e8cdcb62d42317d"}
Jul 16 22:35:18.456963 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:18.456921    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"25373d74-3d14-41e1-a3df-38d3c9d1723b","Type":"ContainerStarted","Data":"c18d71c5f8120c0a864124b18fa0a84f784a0a5061ee63b85a5f281b80dc9530"}
Jul 16 22:35:18.483739 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:18.483679    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.390316728 podStartE2EDuration="6.483662265s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.093345537s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:18.481103245 +0000 UTC m=+2729.460452543" watchObservedRunningTime="2026-07-16 22:35:18.483662265 +0000 UTC m=+2729.463011563"
Jul 16 22:35:19.147517 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:19.147481    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:20.635142 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:20.635102    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:35:23.147606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:23.147556    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:35:23.149566 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:23.149527    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:35:30.634986 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:30.634939    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:35:33.148343 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:33.148296    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:35:40.635582 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:40.635511    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:35:43.147648 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:43.147597    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:35:50.634922 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:50.634864    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:35:53.148323 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:35:53.148275    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:36:00.634834 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:00.634762    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:36:03.147639 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:03.147586    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:36:10.635674 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:10.635566    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:36:13.148320 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:13.148271    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:36:20.635624 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:20.635568    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:36:23.147987 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:23.147940    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:36:30.645386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:30.645337    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:36:30.654720 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:30.654687    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:36:33.148350 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:33.148311    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:36:43.148468 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:43.148422    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:36:53.148195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:36:53.148131    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:37:03.147723 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:03.147679    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:37:13.148192 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:13.148145    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:37:23.163447 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:23.163397    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:37:23.184351 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:23.184307    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:37:30.505542 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:30.505507    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"]
Jul 16 22:37:35.631325 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:35.631290    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 22:37:45.530280 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:45.530208    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" containerID="cri-o://6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320" gracePeriod=45
Jul 16 22:37:45.646491 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:45.646434    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 16 22:37:46.193363 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.193337    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:37:46.290544 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290462    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tmp-dir\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.290544 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290505    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6ftl\" (UniqueName: \"kubernetes.io/projected/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kube-api-access-v6ftl\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.290792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290559    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kserve-provision-location\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.290792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290588    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tls-certs\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.290792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290634    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-cabundle-cert\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.290792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290766    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-model-cache\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.291034 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290809    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-home\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.291034 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.290860    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-dshm\") pod \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\" (UID: \"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0\") "
Jul 16 22:37:46.291146 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.291071    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:46.291220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.291192    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-cabundle-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.291220 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.291194    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-model-cache" (OuterVolumeSpecName: "model-cache") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:46.291453 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.291369    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-home" (OuterVolumeSpecName: "home") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:46.292978 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.292951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kube-api-access-v6ftl" (OuterVolumeSpecName: "kube-api-access-v6ftl") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "kube-api-access-v6ftl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:37:46.293110 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.293091    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-dshm" (OuterVolumeSpecName: "dshm") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:46.293305 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.293289    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:37:46.310122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.310095    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:46.360806 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.360763    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" (UID: "87b6f5f7-1408-43d2-aa3a-10f068a6bfa0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:46.374208 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.374180    2557 generic.go:356] "Generic (PLEG): container finished" podID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerID="6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320" exitCode=0
Jul 16 22:37:46.374324 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.374237    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"
Jul 16 22:37:46.374324 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.374236    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" event={"ID":"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0","Type":"ContainerDied","Data":"6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320"}
Jul 16 22:37:46.374441 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.374338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw" event={"ID":"87b6f5f7-1408-43d2-aa3a-10f068a6bfa0","Type":"ContainerDied","Data":"d964fc44066bb504164ce616cb234452d762c79745a463a0a8d206a95d341126"}
Jul 16 22:37:46.374441 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.374373    2557 scope.go:117] "RemoveContainer" containerID="6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320"
Jul 16 22:37:46.383800 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.383785    2557 scope.go:117] "RemoveContainer" containerID="9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d"
Jul 16 22:37:46.391705 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391686    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.391784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391708    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.391784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391719    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.391784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391727    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.391784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391733    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.391784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391740    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.391784 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391748    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ftl\" (UniqueName: \"kubernetes.io/projected/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0-kube-api-access-v6ftl\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:46.392032 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.391991    2557 scope.go:117] "RemoveContainer" containerID="6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320"
Jul 16 22:37:46.392228 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:37:46.392212    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320\": container with ID starting with 6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320 not found: ID does not exist" containerID="6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320"
Jul 16 22:37:46.392273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.392235    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320"} err="failed to get container status \"6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320\": rpc error: code = NotFound desc = could not find container \"6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320\": container with ID starting with 6ff1ae9b53ece8f9c08612df6be701a7525f51e127ca3c0993481d837ddf6320 not found: ID does not exist"
Jul 16 22:37:46.392273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.392249    2557 scope.go:117] "RemoveContainer" containerID="9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d"
Jul 16 22:37:46.392490 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:37:46.392474    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d\": container with ID starting with 9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d not found: ID does not exist" containerID="9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d"
Jul 16 22:37:46.392534 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.392495    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d"} err="failed to get container status \"9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d\": rpc error: code = NotFound desc = could not find container \"9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d\": container with ID starting with 9ddd2b3314d1c7f17cf5125f1a69211f987ddd6409fe62c30a6501670661174d not found: ID does not exist"
Jul 16 22:37:46.397018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.396997    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"]
Jul 16 22:37:46.402800 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:46.402781    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-786667d575-d9ffw"]
Jul 16 22:37:47.562792 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:47.562746    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" path="/var/lib/kubelet/pods/87b6f5f7-1408-43d2-aa3a-10f068a6bfa0/volumes"
Jul 16 22:37:50.655410 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:50.655361    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" containerID="cri-o://c18d71c5f8120c0a864124b18fa0a84f784a0a5061ee63b85a5f281b80dc9530" gracePeriod=45
Jul 16 22:37:51.164618 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.164585    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 16 22:37:51.404386 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.404357    2557 generic.go:356] "Generic (PLEG): container finished" podID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerID="c18d71c5f8120c0a864124b18fa0a84f784a0a5061ee63b85a5f281b80dc9530" exitCode=0
Jul 16 22:37:51.404533 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.404432    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"25373d74-3d14-41e1-a3df-38d3c9d1723b","Type":"ContainerDied","Data":"c18d71c5f8120c0a864124b18fa0a84f784a0a5061ee63b85a5f281b80dc9530"}
Jul 16 22:37:51.404533 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.404495    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"25373d74-3d14-41e1-a3df-38d3c9d1723b","Type":"ContainerDied","Data":"98d8e58339cb651d59f7b31da9eea7fc2afc44843fd9a1a680a83f760955a030"}
Jul 16 22:37:51.404533 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.404518    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d8e58339cb651d59f7b31da9eea7fc2afc44843fd9a1a680a83f760955a030"
Jul 16 22:37:51.416153 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.416090    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:37:51.539196 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539140    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-home\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539196 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539182    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25373d74-3d14-41e1-a3df-38d3c9d1723b-tls-certs\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfdk9\" (UniqueName: \"kubernetes.io/projected/25373d74-3d14-41e1-a3df-38d3c9d1723b-kube-api-access-rfdk9\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539253    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-tmp-dir\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539272    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-model-cache\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539301    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-kserve-provision-location\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539463 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539328    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-dshm\") pod \"25373d74-3d14-41e1-a3df-38d3c9d1723b\" (UID: \"25373d74-3d14-41e1-a3df-38d3c9d1723b\") "
Jul 16 22:37:51.539724 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539623    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-model-cache" (OuterVolumeSpecName: "model-cache") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:51.539918 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.539888    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-home" (OuterVolumeSpecName: "home") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:51.542470 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.542442    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25373d74-3d14-41e1-a3df-38d3c9d1723b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:37:51.542599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.542471    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-dshm" (OuterVolumeSpecName: "dshm") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:51.542599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.542489    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25373d74-3d14-41e1-a3df-38d3c9d1723b-kube-api-access-rfdk9" (OuterVolumeSpecName: "kube-api-access-rfdk9") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "kube-api-access-rfdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:37:51.552955 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.552923    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:51.604863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.604785    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25373d74-3d14-41e1-a3df-38d3c9d1723b" (UID: "25373d74-3d14-41e1-a3df-38d3c9d1723b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:51.640249 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640213    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.640249 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640247    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25373d74-3d14-41e1-a3df-38d3c9d1723b-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.640249 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640257    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfdk9\" (UniqueName: \"kubernetes.io/projected/25373d74-3d14-41e1-a3df-38d3c9d1723b-kube-api-access-rfdk9\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.640507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640267    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.640507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640275    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.640507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640283    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.640507 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:51.640290    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25373d74-3d14-41e1-a3df-38d3c9d1723b-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:37:52.410285 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:52.410253    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 22:37:52.432780 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:52.432749    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 22:37:52.436794 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:52.436768    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 22:37:53.562612 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:37:53.562580    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" path="/var/lib/kubelet/pods/25373d74-3d14-41e1-a3df-38d3c9d1723b/volumes"
Jul 16 22:38:01.704903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.704871    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"]
Jul 16 22:38:01.705286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705241    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="storage-initializer"
Jul 16 22:38:01.705286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705252    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="storage-initializer"
Jul 16 22:38:01.705286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705273    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main"
Jul 16 22:38:01.705286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705279    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main"
Jul 16 22:38:01.705286 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705288    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="storage-initializer"
Jul 16 22:38:01.705462 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705294    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="storage-initializer"
Jul 16 22:38:01.705462 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705300    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main"
Jul 16 22:38:01.705462 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705305    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main"
Jul 16 22:38:01.705462 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705369    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="25373d74-3d14-41e1-a3df-38d3c9d1723b" containerName="main"
Jul 16 22:38:01.705462 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.705376    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b6f5f7-1408-43d2-aa3a-10f068a6bfa0" containerName="main"
Jul 16 22:38:01.708513 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.708498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.711279 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.711254    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:38:01.711412 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.711317    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 16 22:38:01.711473 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.711454    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 22:38:01.711529 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.711482    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:38:01.711580 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.711485    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:38:01.711947 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.711929    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:38:01.720202 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.720179    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"]
Jul 16 22:38:01.829666 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.829629    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-model-cache\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.829914 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.829762    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.829914 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.829787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpckd\" (UniqueName: \"kubernetes.io/projected/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kube-api-access-vpckd\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.829914 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.829806    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0352040c-b83f-40dc-a0de-b2c2679b4d1f-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.830115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.829968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-home\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.830115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.830008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-dshm\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.830115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.830038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tls-certs\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.830115 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.830077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931419 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-home\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931419 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931421    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-dshm\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tls-certs\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-model-cache\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931549    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931566    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpckd\" (UniqueName: \"kubernetes.io/projected/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kube-api-access-vpckd\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931680 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931585    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0352040c-b83f-40dc-a0de-b2c2679b4d1f-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.931956 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.931900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-home\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.932137 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.932118    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.932464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.932282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0352040c-b83f-40dc-a0de-b2c2679b4d1f-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.932464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.932513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-model-cache\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.932464 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.932532    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.934292 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.934270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tls-certs\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.934394 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.934320    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-dshm\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:01.940016 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:01.939998    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpckd\" (UniqueName: \"kubernetes.io/projected/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kube-api-access-vpckd\") pod \"lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:02.019303 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:02.019207    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:02.167645 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:02.167616    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"]
Jul 16 22:38:02.169566 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:38:02.169539    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0352040c_b83f_40dc_a0de_b2c2679b4d1f.slice/crio-b50782671d45e8ab84d284420bc44e983fcaee6e36cb6700f06b0a09a26a0d38 WatchSource:0}: Error finding container b50782671d45e8ab84d284420bc44e983fcaee6e36cb6700f06b0a09a26a0d38: Status 404 returned error can't find the container with id b50782671d45e8ab84d284420bc44e983fcaee6e36cb6700f06b0a09a26a0d38
Jul 16 22:38:02.462972 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:02.462931    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" event={"ID":"0352040c-b83f-40dc-a0de-b2c2679b4d1f","Type":"ContainerStarted","Data":"0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50"}
Jul 16 22:38:02.463199 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:02.462988    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" event={"ID":"0352040c-b83f-40dc-a0de-b2c2679b4d1f","Type":"ContainerStarted","Data":"b50782671d45e8ab84d284420bc44e983fcaee6e36cb6700f06b0a09a26a0d38"}
Jul 16 22:38:06.487588 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:06.487550    2557 generic.go:356] "Generic (PLEG): container finished" podID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerID="0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50" exitCode=0
Jul 16 22:38:06.488144 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:06.487631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" event={"ID":"0352040c-b83f-40dc-a0de-b2c2679b4d1f","Type":"ContainerDied","Data":"0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50"}
Jul 16 22:38:07.496229 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:07.496194    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" event={"ID":"0352040c-b83f-40dc-a0de-b2c2679b4d1f","Type":"ContainerStarted","Data":"f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0"}
Jul 16 22:38:07.523294 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:07.523237    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podStartSLOduration=2.847439456 podStartE2EDuration="6.523221462s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675782006s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:07.518616281 +0000 UTC m=+2898.497965590" watchObservedRunningTime="2026-07-16 22:38:07.523221462 +0000 UTC m=+2898.502570761"
Jul 16 22:38:08.019602 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:08.019569    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:12.019961 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:12.019923    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:38:12.021536 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:12.021509    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:38:22.020390 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:22.020347    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:38:32.020002 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:32.019948    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:38:42.020231 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:42.020183    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:38:52.019825 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:38:52.019766    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:39:02.020086 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:02.020037    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:39:12.020469 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:12.020381    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:39:22.020285 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:22.020234    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:39:32.030774 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:32.030740    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:39:32.039019 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:32.038989    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:39:49.843630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:49.843602    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:39:49.847007 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:49.846980    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:39:49.849133 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:49.849111    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:39:49.852621 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:39:49.852599    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:40:27.697936 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:27.697902    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"]
Jul 16 22:40:42.721410 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:42.721289    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" containerID="cri-o://f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0" gracePeriod=45
Jul 16 22:40:43.032159 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.032049    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 16 22:40:43.391593 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.391557    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:40:43.415110 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.415076    2557 generic.go:356] "Generic (PLEG): container finished" podID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerID="f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0" exitCode=0
Jul 16 22:40:43.415273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.415166    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" event={"ID":"0352040c-b83f-40dc-a0de-b2c2679b4d1f","Type":"ContainerDied","Data":"f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0"}
Jul 16 22:40:43.415273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.415211    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq" event={"ID":"0352040c-b83f-40dc-a0de-b2c2679b4d1f","Type":"ContainerDied","Data":"b50782671d45e8ab84d284420bc44e983fcaee6e36cb6700f06b0a09a26a0d38"}
Jul 16 22:40:43.415273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.415221    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"
Jul 16 22:40:43.415273 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.415238    2557 scope.go:117] "RemoveContainer" containerID="f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0"
Jul 16 22:40:43.424971 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.424946    2557 scope.go:117] "RemoveContainer" containerID="0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50"
Jul 16 22:40:43.434554 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.434534    2557 scope.go:117] "RemoveContainer" containerID="f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0"
Jul 16 22:40:43.434871 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:40:43.434846    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0\": container with ID starting with f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0 not found: ID does not exist" containerID="f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0"
Jul 16 22:40:43.434964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.434882    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0"} err="failed to get container status \"f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0\": rpc error: code = NotFound desc = could not find container \"f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0\": container with ID starting with f426fe91f1bc7890fda515707f800267e24f550d1bf3a2d4849a9d28ed1134c0 not found: ID does not exist"
Jul 16 22:40:43.434964 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.434908    2557 scope.go:117] "RemoveContainer" containerID="0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50"
Jul 16 22:40:43.435232 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:40:43.435212    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50\": container with ID starting with 0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50 not found: ID does not exist" containerID="0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50"
Jul 16 22:40:43.435309 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.435242    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50"} err="failed to get container status \"0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50\": rpc error: code = NotFound desc = could not find container \"0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50\": container with ID starting with 0a135ac0bae1cb814ae6d1543225fb2ec0680bb063555b136d8886ffaad32b50 not found: ID does not exist"
Jul 16 22:40:43.451705 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451677    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-model-cache\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.451891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451716    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-dshm\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.451891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451778    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tls-certs\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.451891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451800    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tmp-dir\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.451891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451878    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0352040c-b83f-40dc-a0de-b2c2679b4d1f-cabundle-cert\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.452120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kserve-provision-location\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.452120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451959    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-home\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.452120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.451994    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpckd\" (UniqueName: \"kubernetes.io/projected/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kube-api-access-vpckd\") pod \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\" (UID: \"0352040c-b83f-40dc-a0de-b2c2679b4d1f\") "
Jul 16 22:40:43.452120 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.452040    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:43.452332 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.452252    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.452384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.452355    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0352040c-b83f-40dc-a0de-b2c2679b4d1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:43.452443 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.452403    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-home" (OuterVolumeSpecName: "home") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:43.454384 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.454349    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:43.454704 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.454674    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-dshm" (OuterVolumeSpecName: "dshm") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:43.454943 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.454906    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kube-api-access-vpckd" (OuterVolumeSpecName: "kube-api-access-vpckd") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "kube-api-access-vpckd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:43.465722 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.465667    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:43.521777 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.521735    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0352040c-b83f-40dc-a0de-b2c2679b4d1f" (UID: "0352040c-b83f-40dc-a0de-b2c2679b4d1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553416    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0352040c-b83f-40dc-a0de-b2c2679b4d1f-cabundle-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553446    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553457    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553466    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpckd\" (UniqueName: \"kubernetes.io/projected/0352040c-b83f-40dc-a0de-b2c2679b4d1f-kube-api-access-vpckd\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553476    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553485    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.553502 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.553493    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352040c-b83f-40dc-a0de-b2c2679b4d1f-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.737238 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.737202    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"]
Jul 16 22:40:43.742969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:43.742939    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-64fdd68f44-jsxsq"]
Jul 16 22:40:45.563860 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:45.563806    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" path="/var/lib/kubelet/pods/0352040c-b83f-40dc-a0de-b2c2679b4d1f/volumes"
Jul 16 22:40:54.940560 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.940521    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"]
Jul 16 22:40:54.941024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.940885    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="storage-initializer"
Jul 16 22:40:54.941024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.940896    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="storage-initializer"
Jul 16 22:40:54.941024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.940915    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main"
Jul 16 22:40:54.941024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.940920    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main"
Jul 16 22:40:54.941024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.940977    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0352040c-b83f-40dc-a0de-b2c2679b4d1f" containerName="main"
Jul 16 22:40:54.945902 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.945881    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:54.950488 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.950458    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:40:54.950650 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.950489    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:40:54.950650 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.950458    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 16 22:40:54.950877 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.950768    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:40:54.951343 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.951325    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:40:54.958661 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:54.958637    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"]
Jul 16 22:40:55.051405 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051363    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-dshm\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.051405 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tmp-dir\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.051630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-model-cache\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.051630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-home\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.051630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tls-certs\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.051630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051551    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q526l\" (UniqueName: \"kubernetes.io/projected/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kube-api-access-q526l\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.051630 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.051595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kserve-provision-location\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.152863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.152782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q526l\" (UniqueName: \"kubernetes.io/projected/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kube-api-access-q526l\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.152863 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.152841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kserve-provision-location\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.152886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-dshm\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.152912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tmp-dir\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.152935    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-model-cache\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.152963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-home\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.153096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tls-certs\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153381 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.153340    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tmp-dir\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153446 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.153395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-model-cache\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153446 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.153432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kserve-provision-location\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.153539 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.153457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-home\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.155511 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.155492    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-dshm\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.155678 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.155660    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tls-certs\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.164822 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.164766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q526l\" (UniqueName: \"kubernetes.io/projected/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kube-api-access-q526l\") pod \"tls-verification-test-kserve-6bc7757754-k6d85\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.258007 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.257933    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:40:55.405490 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.405465    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"]
Jul 16 22:40:55.407167 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:40:55.407142    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a720e1_e3c9_47c0_ac92_f1ae9b4aa370.slice/crio-7db56898a14f168dcd0b84558e476eb158b4919da54a70b30359f8c9dbc26bdf WatchSource:0}: Error finding container 7db56898a14f168dcd0b84558e476eb158b4919da54a70b30359f8c9dbc26bdf: Status 404 returned error can't find the container with id 7db56898a14f168dcd0b84558e476eb158b4919da54a70b30359f8c9dbc26bdf
Jul 16 22:40:55.487277 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.487243    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" event={"ID":"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370","Type":"ContainerStarted","Data":"dd21fda195d008bd7d1e721e9a2fef57c4ecb1964deed188e907108ce5baadc3"}
Jul 16 22:40:55.487429 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:55.487294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" event={"ID":"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370","Type":"ContainerStarted","Data":"7db56898a14f168dcd0b84558e476eb158b4919da54a70b30359f8c9dbc26bdf"}
Jul 16 22:40:58.503951 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:58.503918    2557 generic.go:356] "Generic (PLEG): container finished" podID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerID="dd21fda195d008bd7d1e721e9a2fef57c4ecb1964deed188e907108ce5baadc3" exitCode=0
Jul 16 22:40:58.504775 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:58.503998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" event={"ID":"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370","Type":"ContainerDied","Data":"dd21fda195d008bd7d1e721e9a2fef57c4ecb1964deed188e907108ce5baadc3"}
Jul 16 22:40:59.511360 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:59.511324    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" event={"ID":"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370","Type":"ContainerStarted","Data":"a070856e4905466095ffe3a51d4607b8c0d413f87b6e4b8b76acd7bb25d3c743"}
Jul 16 22:40:59.546437 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:40:59.546366    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podStartSLOduration=2.753350212 podStartE2EDuration="5.546345693s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.792995481s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:59.539443204 +0000 UTC m=+3070.518792514" watchObservedRunningTime="2026-07-16 22:40:59.546345693 +0000 UTC m=+3070.525694992"
Jul 16 22:41:00.259168 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:00.259121    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:41:05.258636 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:05.258592    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:41:05.262119 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:05.260442    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:41:15.259088 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:15.259038    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:41:25.259419 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:25.259372    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:41:35.258884 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:35.258832    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:41:44.142258 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.142216    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"]
Jul 16 22:41:44.145606 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.145582    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.148472 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.148453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hzzm6\""
Jul 16 22:41:44.149339 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.149319    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 16 22:41:44.161328 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.161300    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"]
Jul 16 22:41:44.288992 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.288952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2678f4c1-566f-480d-884b-db87605b40ad-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.289165 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.289018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678f4c1-566f-480d-884b-db87605b40ad-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.289165 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.289136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2f7\" (UniqueName: \"kubernetes.io/projected/2678f4c1-566f-480d-884b-db87605b40ad-kube-api-access-2m2f7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.390580 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.390545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2678f4c1-566f-480d-884b-db87605b40ad-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.390580 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.390579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678f4c1-566f-480d-884b-db87605b40ad-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.390798 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.390639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2f7\" (UniqueName: \"kubernetes.io/projected/2678f4c1-566f-480d-884b-db87605b40ad-kube-api-access-2m2f7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.391044 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.391021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2678f4c1-566f-480d-884b-db87605b40ad-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.393306 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.393246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678f4c1-566f-480d-884b-db87605b40ad-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.408334 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.408303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2f7\" (UniqueName: \"kubernetes.io/projected/2678f4c1-566f-480d-884b-db87605b40ad-kube-api-access-2m2f7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.455125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.455098    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.611500 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.611469    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"]
Jul 16 22:41:44.611982 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:41:44.611960    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2678f4c1_566f_480d_884b_db87605b40ad.slice/crio-3128138fffb7992961ae7ab58315821862c82b7fe620a94bf4e9a1cd27a717ad WatchSource:0}: Error finding container 3128138fffb7992961ae7ab58315821862c82b7fe620a94bf4e9a1cd27a717ad: Status 404 returned error can't find the container with id 3128138fffb7992961ae7ab58315821862c82b7fe620a94bf4e9a1cd27a717ad
Jul 16 22:41:44.764714 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.764678    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" event={"ID":"2678f4c1-566f-480d-884b-db87605b40ad","Type":"ContainerStarted","Data":"40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28"}
Jul 16 22:41:44.764904 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.764727    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" event={"ID":"2678f4c1-566f-480d-884b-db87605b40ad","Type":"ContainerStarted","Data":"3128138fffb7992961ae7ab58315821862c82b7fe620a94bf4e9a1cd27a717ad"}
Jul 16 22:41:44.764969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.764919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:41:44.788955 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:44.788902    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" podStartSLOduration=0.788886215 podStartE2EDuration="788.886215ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:44.788059494 +0000 UTC m=+3115.767408793" watchObservedRunningTime="2026-07-16 22:41:44.788886215 +0000 UTC m=+3115.768235514"
Jul 16 22:41:45.259483 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:45.259430    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:41:50.016137 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:50.016106    2557 scope.go:117] "RemoveContainer" containerID="c18d71c5f8120c0a864124b18fa0a84f784a0a5061ee63b85a5f281b80dc9530"
Jul 16 22:41:50.026929 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:50.026907    2557 scope.go:117] "RemoveContainer" containerID="b7f8624e6969d92fc14cab3638b230132abcf72205a0d9632e8cdcb62d42317d"
Jul 16 22:41:55.259000 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:41:55.258952    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:42:05.258662 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:05.258616    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:42:15.258799 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:15.258708    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:42:15.772484 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:15.772444    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:42:18.429631 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:18.429592    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"]
Jul 16 22:42:18.431618 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:18.431574    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:42:25.259138 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:25.259084    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:42:28.431611 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:28.431567    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:42:33.451246 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.451197    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main" containerID="cri-o://40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28" gracePeriod=45
Jul 16 22:42:33.698658 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.698624    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:42:33.842343 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.842252    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m2f7\" (UniqueName: \"kubernetes.io/projected/2678f4c1-566f-480d-884b-db87605b40ad-kube-api-access-2m2f7\") pod \"2678f4c1-566f-480d-884b-db87605b40ad\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") "
Jul 16 22:42:33.842343 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.842320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678f4c1-566f-480d-884b-db87605b40ad-tls-certs\") pod \"2678f4c1-566f-480d-884b-db87605b40ad\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") "
Jul 16 22:42:33.842580 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.842357    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2678f4c1-566f-480d-884b-db87605b40ad-tokenizer-uds\") pod \"2678f4c1-566f-480d-884b-db87605b40ad\" (UID: \"2678f4c1-566f-480d-884b-db87605b40ad\") "
Jul 16 22:42:33.842685 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.842660    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678f4c1-566f-480d-884b-db87605b40ad-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2678f4c1-566f-480d-884b-db87605b40ad" (UID: "2678f4c1-566f-480d-884b-db87605b40ad"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:33.844669 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.844638    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2678f4c1-566f-480d-884b-db87605b40ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2678f4c1-566f-480d-884b-db87605b40ad" (UID: "2678f4c1-566f-480d-884b-db87605b40ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:42:33.844800 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.844748    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2678f4c1-566f-480d-884b-db87605b40ad-kube-api-access-2m2f7" (OuterVolumeSpecName: "kube-api-access-2m2f7") pod "2678f4c1-566f-480d-884b-db87605b40ad" (UID: "2678f4c1-566f-480d-884b-db87605b40ad"). InnerVolumeSpecName "kube-api-access-2m2f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:42:33.943717 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.943678    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2678f4c1-566f-480d-884b-db87605b40ad-tokenizer-uds\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:42:33.943920 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.943729    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2f7\" (UniqueName: \"kubernetes.io/projected/2678f4c1-566f-480d-884b-db87605b40ad-kube-api-access-2m2f7\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:42:33.943920 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:33.943740    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678f4c1-566f-480d-884b-db87605b40ad-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:42:34.046413 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.046373    2557 generic.go:356] "Generic (PLEG): container finished" podID="2678f4c1-566f-480d-884b-db87605b40ad" containerID="40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28" exitCode=0
Jul 16 22:42:34.046583 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.046425    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"
Jul 16 22:42:34.046583 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.046453    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" event={"ID":"2678f4c1-566f-480d-884b-db87605b40ad","Type":"ContainerDied","Data":"40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28"}
Jul 16 22:42:34.046583 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.046529    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct" event={"ID":"2678f4c1-566f-480d-884b-db87605b40ad","Type":"ContainerDied","Data":"3128138fffb7992961ae7ab58315821862c82b7fe620a94bf4e9a1cd27a717ad"}
Jul 16 22:42:34.046583 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.046551    2557 scope.go:117] "RemoveContainer" containerID="40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28"
Jul 16 22:42:34.056726 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.056711    2557 scope.go:117] "RemoveContainer" containerID="40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28"
Jul 16 22:42:34.057039 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:42:34.057015    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28\": container with ID starting with 40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28 not found: ID does not exist" containerID="40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28"
Jul 16 22:42:34.057113 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.057047    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28"} err="failed to get container status \"40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28\": rpc error: code = NotFound desc = could not find container \"40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28\": container with ID starting with 40867854eaad809132e8d46074008a48f0ebce015d53ce8c8a3ed73226720f28 not found: ID does not exist"
Jul 16 22:42:34.071590 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.071554    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"]
Jul 16 22:42:34.073688 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:34.073661    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85f6mqct"]
Jul 16 22:42:35.258854 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:35.258766    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:42:35.563876 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:35.563758    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2678f4c1-566f-480d-884b-db87605b40ad" path="/var/lib/kubelet/pods/2678f4c1-566f-480d-884b-db87605b40ad/volumes"
Jul 16 22:42:45.258965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:45.258914    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 16 22:42:55.268461 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:55.268424    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:42:55.276337 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:55.276307    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:42:59.641919 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:42:59.641877    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"]
Jul 16 22:43:14.666140 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:14.666080    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" containerID="cri-o://a070856e4905466095ffe3a51d4607b8c0d413f87b6e4b8b76acd7bb25d3c743" gracePeriod=45
Jul 16 22:43:15.265651 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.265616    2557 generic.go:356] "Generic (PLEG): container finished" podID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerID="a070856e4905466095ffe3a51d4607b8c0d413f87b6e4b8b76acd7bb25d3c743" exitCode=0
Jul 16 22:43:15.265867 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.265681    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" event={"ID":"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370","Type":"ContainerDied","Data":"a070856e4905466095ffe3a51d4607b8c0d413f87b6e4b8b76acd7bb25d3c743"}
Jul 16 22:43:15.319312 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.319285    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:43:15.405036 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.404974    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tmp-dir\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405036 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q526l\" (UniqueName: \"kubernetes.io/projected/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kube-api-access-q526l\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405072    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tls-certs\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405089    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-dshm\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405104    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-home\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405122    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kserve-provision-location\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-model-cache\") pod \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\" (UID: \"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370\") "
Jul 16 22:43:15.405570 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.405540    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-model-cache" (OuterVolumeSpecName: "model-cache") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:15.406245 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.406210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-home" (OuterVolumeSpecName: "home") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:15.407447 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.407404    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kube-api-access-q526l" (OuterVolumeSpecName: "kube-api-access-q526l") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "kube-api-access-q526l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:15.407545 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.407525    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:43:15.407706 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.407681    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-dshm" (OuterVolumeSpecName: "dshm") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:15.421000 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.420979    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:15.473055 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.473016    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" (UID: "61a720e1-e3c9-47c0-ac92-f1ae9b4aa370"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:15.506713 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506693    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.506835 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506716    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.506835 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506746    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.506835 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506760    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q526l\" (UniqueName: \"kubernetes.io/projected/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-kube-api-access-q526l\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.506835 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506772    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.506835 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506783    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.506835 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:15.506790    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:43:16.269427 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.269391    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:43:16.272265 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.272247    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"
Jul 16 22:43:16.272348 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.272265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85" event={"ID":"61a720e1-e3c9-47c0-ac92-f1ae9b4aa370","Type":"ContainerDied","Data":"7db56898a14f168dcd0b84558e476eb158b4919da54a70b30359f8c9dbc26bdf"}
Jul 16 22:43:16.272348 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.272315    2557 scope.go:117] "RemoveContainer" containerID="a070856e4905466095ffe3a51d4607b8c0d413f87b6e4b8b76acd7bb25d3c743"
Jul 16 22:43:16.280970 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.280952    2557 scope.go:117] "RemoveContainer" containerID="dd21fda195d008bd7d1e721e9a2fef57c4ecb1964deed188e907108ce5baadc3"
Jul 16 22:43:16.299733 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.299710    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"]
Jul 16 22:43:16.304832 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:16.304796    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6bc7757754-k6d85"]
Jul 16 22:43:17.562299 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:17.562265    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" path="/var/lib/kubelet/pods/61a720e1-e3c9-47c0-ac92-f1ae9b4aa370/volumes"
Jul 16 22:43:24.292672 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.292642    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"]
Jul 16 22:43:24.293122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293030    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main"
Jul 16 22:43:24.293122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293042    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main"
Jul 16 22:43:24.293122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293056    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="storage-initializer"
Jul 16 22:43:24.293122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293061    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="storage-initializer"
Jul 16 22:43:24.293122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293068    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main"
Jul 16 22:43:24.293122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293073    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main"
Jul 16 22:43:24.293339 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293136    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a720e1-e3c9-47c0-ac92-f1ae9b4aa370" containerName="main"
Jul 16 22:43:24.293339 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.293143    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2678f4c1-566f-480d-884b-db87605b40ad" containerName="main"
Jul 16 22:43:24.297899 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.297882    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.300355 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.300330    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:43:24.300490 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.300380    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 16 22:43:24.300694 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.300678    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:43:24.300788 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.300719    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:43:24.300788 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.300774    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:43:24.308700 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.308677    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"]
Jul 16 22:43:24.375289 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-kserve-provision-location\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.375389 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-home\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.375389 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bccc131-e383-4f67-9df5-9534de7fc181-tls-certs\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.375389 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-model-cache\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.375497 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375394    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-tmp-dir\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.375497 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375434    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvd7\" (UniqueName: \"kubernetes.io/projected/3bccc131-e383-4f67-9df5-9534de7fc181-kube-api-access-frvd7\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.375497 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.375451    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-dshm\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476070 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476041    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvd7\" (UniqueName: \"kubernetes.io/projected/3bccc131-e383-4f67-9df5-9534de7fc181-kube-api-access-frvd7\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-dshm\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-kserve-provision-location\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-home\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bccc131-e383-4f67-9df5-9534de7fc181-tls-certs\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476418 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476233    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-model-cache\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476418 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476259    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-tmp-dir\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476573 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-home\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476643 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476605    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-kserve-provision-location\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476714 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476695    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-model-cache\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.476773 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.476733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-tmp-dir\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.478520 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.478501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-dshm\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.478848 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.478830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bccc131-e383-4f67-9df5-9534de7fc181-tls-certs\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.488068 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.488049    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvd7\" (UniqueName: \"kubernetes.io/projected/3bccc131-e383-4f67-9df5-9534de7fc181-kube-api-access-frvd7\") pod \"prestop-hook-test-kserve-6785946898-6ph2c\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.608280 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.608231    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:24.752710 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:24.752685    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"]
Jul 16 22:43:24.754602 ip-10-0-136-80 kubenswrapper[2557]: W0716 22:43:24.754576    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bccc131_e383_4f67_9df5_9534de7fc181.slice/crio-31d9a9cea9235853ddc29514162e8791dc9dbf857c6e9d442668b3c09d384650 WatchSource:0}: Error finding container 31d9a9cea9235853ddc29514162e8791dc9dbf857c6e9d442668b3c09d384650: Status 404 returned error can't find the container with id 31d9a9cea9235853ddc29514162e8791dc9dbf857c6e9d442668b3c09d384650
Jul 16 22:43:25.319880 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:25.319845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" event={"ID":"3bccc131-e383-4f67-9df5-9534de7fc181","Type":"ContainerStarted","Data":"c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39"}
Jul 16 22:43:25.320238 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:25.319927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" event={"ID":"3bccc131-e383-4f67-9df5-9534de7fc181","Type":"ContainerStarted","Data":"31d9a9cea9235853ddc29514162e8791dc9dbf857c6e9d442668b3c09d384650"}
Jul 16 22:43:28.338473 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:28.338435    2557 generic.go:356] "Generic (PLEG): container finished" podID="3bccc131-e383-4f67-9df5-9534de7fc181" containerID="c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39" exitCode=0
Jul 16 22:43:28.339121 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:28.338497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" event={"ID":"3bccc131-e383-4f67-9df5-9534de7fc181","Type":"ContainerDied","Data":"c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39"}
Jul 16 22:43:29.345883 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:29.345844    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" event={"ID":"3bccc131-e383-4f67-9df5-9534de7fc181","Type":"ContainerStarted","Data":"7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492"}
Jul 16 22:43:29.381457 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:29.381404    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podStartSLOduration=2.5601678249999997 podStartE2EDuration="5.381388492s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.821220667s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:29.377196835 +0000 UTC m=+3220.356546134" watchObservedRunningTime="2026-07-16 22:43:29.381388492 +0000 UTC m=+3220.360737791"
Jul 16 22:43:29.609276 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:29.609178    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:34.608430 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:34.608385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:43:34.610028 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:34.609996    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:43:44.609670 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:44.609613    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:43:54.608752 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:43:54.608707    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:44:04.609345 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:04.609284    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:44:14.608656 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:14.608615    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:44:24.609342 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:24.609281    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:44:34.609073 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:34.609026    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:44:44.608827 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:44.608779    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:44:49.884106 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:49.884076    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:44:49.888662 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:49.888637    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:44:49.889402 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:49.889379    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:44:49.894024 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:49.894006    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:44:54.609233 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:44:54.609183    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:45:04.608610 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:04.608569    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:45:14.609425 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:14.609318    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:45:24.619699 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:24.619666    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:45:24.628693 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:24.628665    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:45:25.795269 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.795236    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"]
Jul 16 22:45:25.818162 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.818126    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"]
Jul 16 22:45:25.823449 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.823429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.841032 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.841001    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"]
Jul 16 22:45:25.894765 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.894730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-dshm\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.894965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.894780    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtdc\" (UniqueName: \"kubernetes.io/projected/7901abc5-1ba8-454b-91c4-d44d15f96208-kube-api-access-tvtdc\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.894965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.894861    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-home\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.894965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.894899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7901abc5-1ba8-454b-91c4-d44d15f96208-tls-certs\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.894965 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.894947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-kserve-provision-location\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.895109 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.894974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-tmp-dir\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.895109 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.895014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-model-cache\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995442 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995394    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-kserve-provision-location\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995649 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995452    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-tmp-dir\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995649 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-model-cache\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995649 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-dshm\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995649 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtdc\" (UniqueName: \"kubernetes.io/projected/7901abc5-1ba8-454b-91c4-d44d15f96208-kube-api-access-tvtdc\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995649 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-home\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.995649 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7901abc5-1ba8-454b-91c4-d44d15f96208-tls-certs\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.996002 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-kserve-provision-location\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.996002 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-model-cache\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.996002 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.995965    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-tmp-dir\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.996149 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.996020    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-home\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.998060 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.998035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-dshm\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:25.998331 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:25.998311    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7901abc5-1ba8-454b-91c4-d44d15f96208-tls-certs\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:26.004408 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:26.004388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtdc\" (UniqueName: \"kubernetes.io/projected/7901abc5-1ba8-454b-91c4-d44d15f96208-kube-api-access-tvtdc\") pod \"prestop-hook-test-kserve-6785946898-2mr2b\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:26.133356 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:26.133315    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:26.281653 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:26.281621    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"]
Jul 16 22:45:27.027308 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:27.027271    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" event={"ID":"7901abc5-1ba8-454b-91c4-d44d15f96208","Type":"ContainerStarted","Data":"e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed"}
Jul 16 22:45:27.027675 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:27.027326    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" event={"ID":"7901abc5-1ba8-454b-91c4-d44d15f96208","Type":"ContainerStarted","Data":"6c1a4713032e32263c395a885d35be0e66447eb9f91c3189c1f107bd07015c61"}
Jul 16 22:45:30.049169 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:30.049134    2557 generic.go:356] "Generic (PLEG): container finished" podID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerID="e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed" exitCode=0
Jul 16 22:45:30.049567 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:30.049210    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" event={"ID":"7901abc5-1ba8-454b-91c4-d44d15f96208","Type":"ContainerDied","Data":"e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed"}
Jul 16 22:45:31.057685 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:31.057649    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" event={"ID":"7901abc5-1ba8-454b-91c4-d44d15f96208","Type":"ContainerStarted","Data":"86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f"}
Jul 16 22:45:31.085015 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:31.084966    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" podStartSLOduration=3.095425355 podStartE2EDuration="6.084948578s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.989523223s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:31.083297639 +0000 UTC m=+3342.062646937" watchObservedRunningTime="2026-07-16 22:45:31.084948578 +0000 UTC m=+3342.064297876"
Jul 16 22:45:31.133664 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:31.133634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:36.133647 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:36.133608    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:45:36.135246 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:36.135217    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 16 22:45:41.047936 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.047893    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" containerID="cri-o://7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492" gracePeriod=45
Jul 16 22:45:41.621153 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.621111    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 16 22:45:41.811500 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.811471    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:45:41.935864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.935809    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-tmp-dir\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.935864 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.935869    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-kserve-provision-location\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.936125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.935886    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-dshm\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.936125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.935931    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frvd7\" (UniqueName: \"kubernetes.io/projected/3bccc131-e383-4f67-9df5-9534de7fc181-kube-api-access-frvd7\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.936125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.935963    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-home\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.936125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.936008    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-model-cache\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.936125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.936052    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bccc131-e383-4f67-9df5-9534de7fc181-tls-certs\") pod \"3bccc131-e383-4f67-9df5-9534de7fc181\" (UID: \"3bccc131-e383-4f67-9df5-9534de7fc181\") "
Jul 16 22:45:41.936519 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.936455    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-model-cache" (OuterVolumeSpecName: "model-cache") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:41.936602 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.936580    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-home" (OuterVolumeSpecName: "home") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:41.938520 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.938492    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-dshm" (OuterVolumeSpecName: "dshm") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:41.938736 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.938708    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bccc131-e383-4f67-9df5-9534de7fc181-kube-api-access-frvd7" (OuterVolumeSpecName: "kube-api-access-frvd7") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "kube-api-access-frvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:45:41.938906 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.938858    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bccc131-e383-4f67-9df5-9534de7fc181-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:45:41.955157 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:41.955121    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:42.008391 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.008348    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bccc131-e383-4f67-9df5-9534de7fc181" (UID: "3bccc131-e383-4f67-9df5-9534de7fc181"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:42.037125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037096    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bccc131-e383-4f67-9df5-9534de7fc181-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.037125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037122    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.037125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037132    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.037532 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037142    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.037532 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037154    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvd7\" (UniqueName: \"kubernetes.io/projected/3bccc131-e383-4f67-9df5-9534de7fc181-kube-api-access-frvd7\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.037532 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037161    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.037532 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.037171    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bccc131-e383-4f67-9df5-9534de7fc181-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:45:42.121837 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.121775    2557 generic.go:356] "Generic (PLEG): container finished" podID="3bccc131-e383-4f67-9df5-9534de7fc181" containerID="7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492" exitCode=0
Jul 16 22:45:42.122222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.121840    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"
Jul 16 22:45:42.122222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.121843    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" event={"ID":"3bccc131-e383-4f67-9df5-9534de7fc181","Type":"ContainerDied","Data":"7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492"}
Jul 16 22:45:42.122222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.121967    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c" event={"ID":"3bccc131-e383-4f67-9df5-9534de7fc181","Type":"ContainerDied","Data":"31d9a9cea9235853ddc29514162e8791dc9dbf857c6e9d442668b3c09d384650"}
Jul 16 22:45:42.122222 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.121991    2557 scope.go:117] "RemoveContainer" containerID="7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492"
Jul 16 22:45:42.133425 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.133407    2557 scope.go:117] "RemoveContainer" containerID="c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39"
Jul 16 22:45:42.145772 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.145748    2557 scope.go:117] "RemoveContainer" containerID="7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492"
Jul 16 22:45:42.146063 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:45:42.146043    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492\": container with ID starting with 7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492 not found: ID does not exist" containerID="7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492"
Jul 16 22:45:42.146136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.146073    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492"} err="failed to get container status \"7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492\": rpc error: code = NotFound desc = could not find container \"7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492\": container with ID starting with 7e7cfdbdbc8497f7a6663218031217237ef14cec8dd5cfc9d32658811dba0492 not found: ID does not exist"
Jul 16 22:45:42.146136 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.146089    2557 scope.go:117] "RemoveContainer" containerID="c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39"
Jul 16 22:45:42.146315 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:45:42.146295    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39\": container with ID starting with c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39 not found: ID does not exist" containerID="c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39"
Jul 16 22:45:42.146357 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.146319    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39"} err="failed to get container status \"c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39\": rpc error: code = NotFound desc = could not find container \"c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39\": container with ID starting with c7ca825d8e639c41e73351d9bfb976550b2989eecf305b0458bffa8aa3d2df39 not found: ID does not exist"
Jul 16 22:45:42.164290 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.164265    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"]
Jul 16 22:45:42.168693 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:42.168669    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-6ph2c"]
Jul 16 22:45:43.562116 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:43.562080    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" path="/var/lib/kubelet/pods/3bccc131-e383-4f67-9df5-9534de7fc181/volumes"
Jul 16 22:45:44.441861 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:44.441780    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"]
Jul 16 22:45:59.465961 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:45:59.465921    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="main" containerID="cri-o://86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f" gracePeriod=45
Jul 16 22:46:45.122653 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.122625    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-6785946898-2mr2b_7901abc5-1ba8-454b-91c4-d44d15f96208/main/0.log"
Jul 16 22:46:45.124352 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.124336    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:46:45.184487 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184438    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvtdc\" (UniqueName: \"kubernetes.io/projected/7901abc5-1ba8-454b-91c4-d44d15f96208-kube-api-access-tvtdc\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.184701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184522    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7901abc5-1ba8-454b-91c4-d44d15f96208-tls-certs\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.184701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184553    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-home\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.184701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184583    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-dshm\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.184701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184654    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-tmp-dir\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.184701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184685    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-kserve-provision-location\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.184701 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184704    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-model-cache\") pod \"7901abc5-1ba8-454b-91c4-d44d15f96208\" (UID: \"7901abc5-1ba8-454b-91c4-d44d15f96208\") "
Jul 16 22:46:45.185076 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.184991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-home" (OuterVolumeSpecName: "home") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:45.185190 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.185159    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:45.185237 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.185170    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-model-cache" (OuterVolumeSpecName: "model-cache") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:45.187478 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.187442    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-dshm" (OuterVolumeSpecName: "dshm") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:45.187478 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.187450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7901abc5-1ba8-454b-91c4-d44d15f96208-kube-api-access-tvtdc" (OuterVolumeSpecName: "kube-api-access-tvtdc") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "kube-api-access-tvtdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:46:45.187648 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.187450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7901abc5-1ba8-454b-91c4-d44d15f96208-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:46:45.243806 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.243679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7901abc5-1ba8-454b-91c4-d44d15f96208" (UID: "7901abc5-1ba8-454b-91c4-d44d15f96208"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:45.285909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285865    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-dshm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.285909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285909    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-tmp-dir\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.285909 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285921    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.286138 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285931    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-model-cache\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.286138 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285945    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtdc\" (UniqueName: \"kubernetes.io/projected/7901abc5-1ba8-454b-91c4-d44d15f96208-kube-api-access-tvtdc\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.286138 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285957    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7901abc5-1ba8-454b-91c4-d44d15f96208-tls-certs\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.286138 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.285966    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7901abc5-1ba8-454b-91c4-d44d15f96208-home\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.484324 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.484297    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-6785946898-2mr2b_7901abc5-1ba8-454b-91c4-d44d15f96208/main/0.log"
Jul 16 22:46:45.484663 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.484642    2557 generic.go:356] "Generic (PLEG): container finished" podID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerID="86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f" exitCode=137
Jul 16 22:46:45.484781 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.484677    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" event={"ID":"7901abc5-1ba8-454b-91c4-d44d15f96208","Type":"ContainerDied","Data":"86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f"}
Jul 16 22:46:45.484781 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.484700    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"
Jul 16 22:46:45.484781 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.484730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b" event={"ID":"7901abc5-1ba8-454b-91c4-d44d15f96208","Type":"ContainerDied","Data":"6c1a4713032e32263c395a885d35be0e66447eb9f91c3189c1f107bd07015c61"}
Jul 16 22:46:45.484781 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.484764    2557 scope.go:117] "RemoveContainer" containerID="86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f"
Jul 16 22:46:45.498417 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.498398    2557 scope.go:117] "RemoveContainer" containerID="e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed"
Jul 16 22:46:45.507209 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.507185    2557 scope.go:117] "RemoveContainer" containerID="86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f"
Jul 16 22:46:45.507546 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:46:45.507523    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f\": container with ID starting with 86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f not found: ID does not exist" containerID="86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f"
Jul 16 22:46:45.507625 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.507554    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f"} err="failed to get container status \"86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f\": rpc error: code = NotFound desc = could not find container \"86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f\": container with ID starting with 86e4e0ba02785748e0f97090ace5e9e3d2695013f04d43c842d38656fb9ed32f not found: ID does not exist"
Jul 16 22:46:45.507625 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.507572    2557 scope.go:117] "RemoveContainer" containerID="e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed"
Jul 16 22:46:45.507887 ip-10-0-136-80 kubenswrapper[2557]: E0716 22:46:45.507863    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed\": container with ID starting with e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed not found: ID does not exist" containerID="e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed"
Jul 16 22:46:45.507969 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.507900    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed"} err="failed to get container status \"e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed\": rpc error: code = NotFound desc = could not find container \"e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed\": container with ID starting with e180b7acc6484989a89f7cd4ce7fad3cf0df36b8a51f714cee653009d48641ed not found: ID does not exist"
Jul 16 22:46:45.510017 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.509991    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"]
Jul 16 22:46:45.513380 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.513358    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6785946898-2mr2b"]
Jul 16 22:46:45.562104 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:46:45.562072    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" path="/var/lib/kubelet/pods/7901abc5-1ba8-454b-91c4-d44d15f96208/volumes"
Jul 16 22:48:46.513158 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:46.513127    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wgrrr_e4700a47-8782-4770-be87-5f9c000ef5a1/discovery/0.log"
Jul 16 22:48:46.593996 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:46.593963    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-559484955d-q46dq_ec73e762-8ce7-42b8-b8b4-bad1d863c480/router/0.log"
Jul 16 22:48:48.068740 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:48.068706    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wgrrr_e4700a47-8782-4770-be87-5f9c000ef5a1/discovery/0.log"
Jul 16 22:48:48.163575 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:48.163543    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-559484955d-q46dq_ec73e762-8ce7-42b8-b8b4-bad1d863c480/router/0.log"
Jul 16 22:48:49.197599 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:49.197567    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-76qtk_23f107ac-7611-4768-b8c8-6e92eebead92/manager/0.log"
Jul 16 22:48:49.213592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:49.213564    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-6vqdx_c484bc34-bb8e-475a-b29b-8afc25416370/kuadrant-console-plugin/0.log"
Jul 16 22:48:49.413622 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:49.413591    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-kp78l_8548fc5b-22e5-40d5-9ff9-562ead763bd5/manager/0.log"
Jul 16 22:48:49.425352 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:49.425330    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-4xqs4_da4478a9-b81e-4d50-a483-b876b7b90147/limitador/0.log"
Jul 16 22:48:49.441703 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:49.441667    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-ct7rc_656c1e93-3a69-4ca6-a032-d0c8c1706714/manager/0.log"
Jul 16 22:48:52.039579 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.039549    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5v9ch/must-gather-kkrdb"]
Jul 16 22:48:52.040109 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040091    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="main"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040112    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="main"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040126    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040133    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040145    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="storage-initializer"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040154    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="storage-initializer"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040185    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="storage-initializer"
Jul 16 22:48:52.040195 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040193    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="storage-initializer"
Jul 16 22:48:52.040541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040295    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bccc131-e383-4f67-9df5-9534de7fc181" containerName="main"
Jul 16 22:48:52.040541 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.040307    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7901abc5-1ba8-454b-91c4-d44d15f96208" containerName="main"
Jul 16 22:48:52.042489 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.042469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.044911 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.044889    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5v9ch\"/\"default-dockercfg-9x8pv\""
Jul 16 22:48:52.045966 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.045947    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v9ch\"/\"kube-root-ca.crt\""
Jul 16 22:48:52.046037 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.045947    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v9ch\"/\"openshift-service-ca.crt\""
Jul 16 22:48:52.052719 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.052696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v9ch/must-gather-kkrdb"]
Jul 16 22:48:52.094021 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.093986    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a20955d-23ce-4ea2-a1fe-05edbab6bb1a-must-gather-output\") pod \"must-gather-kkrdb\" (UID: \"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a\") " pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.094198 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.094027    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6x7\" (UniqueName: \"kubernetes.io/projected/6a20955d-23ce-4ea2-a1fe-05edbab6bb1a-kube-api-access-kz6x7\") pod \"must-gather-kkrdb\" (UID: \"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a\") " pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.194854 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.194797    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a20955d-23ce-4ea2-a1fe-05edbab6bb1a-must-gather-output\") pod \"must-gather-kkrdb\" (UID: \"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a\") " pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.194854 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.194858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6x7\" (UniqueName: \"kubernetes.io/projected/6a20955d-23ce-4ea2-a1fe-05edbab6bb1a-kube-api-access-kz6x7\") pod \"must-gather-kkrdb\" (UID: \"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a\") " pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.195177 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.195154    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a20955d-23ce-4ea2-a1fe-05edbab6bb1a-must-gather-output\") pod \"must-gather-kkrdb\" (UID: \"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a\") " pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.203483 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.203461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6x7\" (UniqueName: \"kubernetes.io/projected/6a20955d-23ce-4ea2-a1fe-05edbab6bb1a-kube-api-access-kz6x7\") pod \"must-gather-kkrdb\" (UID: \"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a\") " pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.351734 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.351657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v9ch/must-gather-kkrdb"
Jul 16 22:48:52.495211 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.493887    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v9ch/must-gather-kkrdb"]
Jul 16 22:48:52.500137 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:52.500115    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:48:53.131877 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:53.131837    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/must-gather-kkrdb" event={"ID":"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a","Type":"ContainerStarted","Data":"93f7c3a2225a57592f2b15a473ba401b56bf92eeb073b1043c3bb8fc75ec58fc"}
Jul 16 22:48:54.139875 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:54.139837    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/must-gather-kkrdb" event={"ID":"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a","Type":"ContainerStarted","Data":"7d3638484224ddbc27d60a1fbbbd5af999541835b090a7682c47cedbaca95d12"}
Jul 16 22:48:54.140266 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:54.139890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/must-gather-kkrdb" event={"ID":"6a20955d-23ce-4ea2-a1fe-05edbab6bb1a","Type":"ContainerStarted","Data":"59ccb9521fa526e574bb01e91dbf4a2b57bfd4ac888be99f13c260d5fea69726"}
Jul 16 22:48:54.160074 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:54.160014    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5v9ch/must-gather-kkrdb" podStartSLOduration=1.351098671 podStartE2EDuration="2.159994868s" totalImagesPullingTime="808.896197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:54.15795207 +0000 UTC m=+3545.137301370" watchObservedRunningTime="2026-07-16 22:48:54.159994868 +0000 UTC m=+3545.139344169"
Jul 16 22:48:54.891523 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:54.891461    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cpdkm_4f90b1b1-28ae-4fda-972d-b2be4f1ae0c2/global-pull-secret-syncer/0.log"
Jul 16 22:48:55.075299 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:55.075267    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q8bwn_fba83155-ef2f-4233-a21b-155bb4abf982/konnectivity-agent/0.log"
Jul 16 22:48:55.126577 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:48:55.126540    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-80.ec2.internal_6337947e1d545ee6d7cb3a765fe8fb4c/haproxy/0.log"
Jul 16 22:49:00.161528 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:00.161494    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-76qtk_23f107ac-7611-4768-b8c8-6e92eebead92/manager/0.log"
Jul 16 22:49:00.215189 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:00.215155    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-6vqdx_c484bc34-bb8e-475a-b29b-8afc25416370/kuadrant-console-plugin/0.log"
Jul 16 22:49:00.445890 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:00.444898    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-kp78l_8548fc5b-22e5-40d5-9ff9-562ead763bd5/manager/0.log"
Jul 16 22:49:00.481642 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:00.481340    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-4xqs4_da4478a9-b81e-4d50-a483-b876b7b90147/limitador/0.log"
Jul 16 22:49:00.525229 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:00.525196    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-ct7rc_656c1e93-3a69-4ca6-a032-d0c8c1706714/manager/0.log"
Jul 16 22:49:01.788503 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.788474    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/alertmanager/0.log"
Jul 16 22:49:01.812261 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.812230    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/config-reloader/0.log"
Jul 16 22:49:01.835796 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.835763    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/kube-rbac-proxy-web/0.log"
Jul 16 22:49:01.860135 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.860108    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/kube-rbac-proxy/0.log"
Jul 16 22:49:01.883456 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.883428    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/kube-rbac-proxy-metric/0.log"
Jul 16 22:49:01.907019 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.906978    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/prom-label-proxy/0.log"
Jul 16 22:49:01.932447 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.932422    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55099ce9-fbdd-4d73-9d22-9b95cf5700a2/init-config-reloader/0.log"
Jul 16 22:49:01.990476 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:01.990424    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/1.log"
Jul 16 22:49:02.158903 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:02.158858    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-nkrfq_37dae182-a8ff-48b8-898e-155bda8bf0c5/cluster-monitoring-operator/0.log"
Jul 16 22:49:02.372891 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:02.372859    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d26k5_e792edf0-10d2-409d-9702-ef265c84ba4b/node-exporter/0.log"
Jul 16 22:49:02.398421 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:02.398382    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d26k5_e792edf0-10d2-409d-9702-ef265c84ba4b/kube-rbac-proxy/0.log"
Jul 16 22:49:02.434391 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:02.434304    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d26k5_e792edf0-10d2-409d-9702-ef265c84ba4b/init-textfile/0.log"
Jul 16 22:49:03.033100 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:03.033070    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-856bc5cddb-psgm6_204de874-c30e-409f-849f-2240ca82c63b/telemeter-client/0.log"
Jul 16 22:49:03.057017 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:03.056985    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-856bc5cddb-psgm6_204de874-c30e-409f-849f-2240ca82c63b/reload/0.log"
Jul 16 22:49:03.084683 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:03.084650    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-856bc5cddb-psgm6_204de874-c30e-409f-849f-2240ca82c63b/kube-rbac-proxy/0.log"
Jul 16 22:49:04.787455 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.787427    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/1.log"
Jul 16 22:49:04.798018 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.797994    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t7tsg_5af2916e-742e-4253-a404-425b475b1863/console-operator/2.log"
Jul 16 22:49:04.798255 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.798194    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"]
Jul 16 22:49:04.802543 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.802522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:04.808125 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.808105    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"]
Jul 16 22:49:04.925639 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.925595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-lib-modules\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:04.925860 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.925648    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-proc\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:04.925860 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.925705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-podres\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:04.925860 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.925762    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-sys\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:04.925860 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:04.925801    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbb6\" (UniqueName: \"kubernetes.io/projected/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-kube-api-access-znbb6\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026529 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-lib-modules\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026529 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-proc\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026600    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-proc\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-podres\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-sys\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbb6\" (UniqueName: \"kubernetes.io/projected/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-kube-api-access-znbb6\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-lib-modules\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.026787 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-podres\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.027217 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.026788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-sys\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.035049 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.035026    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbb6\" (UniqueName: \"kubernetes.io/projected/1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2-kube-api-access-znbb6\") pod \"perf-node-gather-daemonset-t78zn\" (UID: \"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.114977 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.114898    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:05.312350 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.312324    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"]
Jul 16 22:49:05.946007 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:05.945974    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-scjs9_a3d0824f-6316-40f5-8268-d138a1d9aa61/volume-data-source-validator/0.log"
Jul 16 22:49:06.262943 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:06.262850    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn" event={"ID":"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2","Type":"ContainerStarted","Data":"f47bf33ecfaff666a46d4a72854b09eff2b8fb3eafacb22ba2f67b2a31aa9d86"}
Jul 16 22:49:06.262943 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:06.262915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn" event={"ID":"1e1d3834-da3c-4ff6-8e33-05ca38a1d1a2","Type":"ContainerStarted","Data":"682f00001c05217501e870595aa37232b264fb4b5ee8a1a4889344468376d6c4"}
Jul 16 22:49:06.263154 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:06.263130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:06.281122 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:06.281050    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn" podStartSLOduration=2.2810287430000002 podStartE2EDuration="2.281028743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:49:06.280080064 +0000 UTC m=+3557.259429363" watchObservedRunningTime="2026-07-16 22:49:06.281028743 +0000 UTC m=+3557.260378045"
Jul 16 22:49:07.012420 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:07.012393    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9px6_f65c0253-132a-4143-9015-1dd343dd797b/dns/0.log"
Jul 16 22:49:07.035211 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:07.035178    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9px6_f65c0253-132a-4143-9015-1dd343dd797b/kube-rbac-proxy/0.log"
Jul 16 22:49:07.111091 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:07.111051    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jkm6h_fb1570a9-59ea-4345-8408-5a192d1e24c6/dns-node-resolver/0.log"
Jul 16 22:49:07.688592 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:07.688563    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6z6pz_52d062bf-25da-41c5-b1ae-0ecd872eaedf/node-ca/0.log"
Jul 16 22:49:09.331574 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:09.331532    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wgrrr_e4700a47-8782-4770-be87-5f9c000ef5a1/discovery/0.log"
Jul 16 22:49:09.430348 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:09.430320    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-559484955d-q46dq_ec73e762-8ce7-42b8-b8b4-bad1d863c480/router/0.log"
Jul 16 22:49:09.960869 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:09.960835    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8rcrk_97aa7d52-d173-4110-90be-35f610bdda8f/serve-healthcheck-canary/0.log"
Jul 16 22:49:10.444883 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:10.444849    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-t4lc2_3afdced7-a64f-4cac-858a-362b32874bd5/insights-operator/0.log"
Jul 16 22:49:10.446150 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:10.446128    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-t4lc2_3afdced7-a64f-4cac-858a-362b32874bd5/insights-operator/1.log"
Jul 16 22:49:10.475175 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:10.475146    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mh8sz_f06de3dc-1b3f-4823-8c95-9a23dd15889b/kube-rbac-proxy/0.log"
Jul 16 22:49:10.498801 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:10.498766    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mh8sz_f06de3dc-1b3f-4823-8c95-9a23dd15889b/exporter/0.log"
Jul 16 22:49:10.522199 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:10.522175    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mh8sz_f06de3dc-1b3f-4823-8c95-9a23dd15889b/extractor/0.log"
Jul 16 22:49:12.281740 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:12.281698    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-t78zn"
Jul 16 22:49:14.343232 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:14.343170    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-84cf6d888d-rptsm_1afc23cb-0744-4cfd-a0fd-3eeb45dfc56c/manager/0.log"
Jul 16 22:49:15.468989 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:15.468907    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-gdzxl_5cb76a0a-af3e-4721-9a0a-dabca6f53dd7/seaweedfs/0.log"
Jul 16 22:49:16.710707 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:16.710675    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wgrrr_e4700a47-8782-4770-be87-5f9c000ef5a1/discovery/0.log"
Jul 16 22:49:16.784408 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:16.784322    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-559484955d-q46dq_ec73e762-8ce7-42b8-b8b4-bad1d863c480/router/0.log"
Jul 16 22:49:18.405690 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:18.405187    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wgrrr_e4700a47-8782-4770-be87-5f9c000ef5a1/discovery/0.log"
Jul 16 22:49:18.488166 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:18.488131    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-559484955d-q46dq_ec73e762-8ce7-42b8-b8b4-bad1d863c480/router/0.log"
Jul 16 22:49:20.773775 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:20.773750    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tzh8s_db192757-d00d-4f91-9cff-ce4820bdba31/migrator/0.log"
Jul 16 22:49:20.795658 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:20.795635    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tzh8s_db192757-d00d-4f91-9cff-ce4820bdba31/graceful-termination/0.log"
Jul 16 22:49:21.148259 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:21.148230    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-vtvvf_8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c/kube-storage-version-migrator-operator/1.log"
Jul 16 22:49:21.150058 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:21.150034    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-vtvvf_8ff8e0a6-ed66-4e44-9aa9-ff1c2271a73c/kube-storage-version-migrator-operator/0.log"
Jul 16 22:49:22.435121 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.435093    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/kube-multus-additional-cni-plugins/0.log"
Jul 16 22:49:22.455974 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.455953    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/egress-router-binary-copy/0.log"
Jul 16 22:49:22.477595 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.477570    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/cni-plugins/0.log"
Jul 16 22:49:22.498391 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.498372    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/bond-cni-plugin/0.log"
Jul 16 22:49:22.520415 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.520395    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/routeoverride-cni/0.log"
Jul 16 22:49:22.543160 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.543140    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/whereabouts-cni-bincopy/0.log"
Jul 16 22:49:22.565426 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.565388    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m95lw_92fc2188-8ee2-47ca-98bf-a477465b096d/whereabouts-cni/0.log"
Jul 16 22:49:22.614264 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.614235    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hh4jx_e1160e2d-5f23-4b46-8369-316e2749a642/kube-multus/0.log"
Jul 16 22:49:22.709923 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.709761    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dpnxv_24fdc43a-8826-46d0-8c18-ace05937ce99/network-metrics-daemon/0.log"
Jul 16 22:49:22.727236 ip-10-0-136-80 kubenswrapper[2557]: I0716 22:49:22.727206    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dpnxv_24fdc43a-8826-46d0-8c18-ace05937ce99/kube-rbac-proxy/0.log"